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ABSTRACT 


Complex warfighter systems are increasingly required for continuing United States 
dominance, which drives a need for high quality Systems Engineering (SE) processes. A 
System Engineering Health and Visualization (SEHV) capability is needed so that 
leadership can gain insight into potential SE risk areas, allowing them to be proactive 
instead of reactive to issues leading to program failures, thus saving time, effort, and 
costs. This capstone’s purpose was to determine if an automated means of collecting and 
displaying SE data trends is feasible and effective. 

To accomplish this, the team analyzed stakeholder’s requirements and performed 
a literature study on SE leading indicators. Modeling and simulation was performed to 
further analyze these requirements and provide the best means to obtain SE health data 
from Space and Naval Warfare System Center Atlantic (SSC-A). This developed the 
SEHV architecture to include data integration strategy. A conceptual model for the 
SEHV capability was produced along with acquisition strategies and cost estimates. 

The research shows a need to incorporate an automated SEHV system into SSC- 
A’s organization to improve efficiencies in data calls and management insight into the SE 
health of a program. Additionally, the team identified future research requirements and 
provided recommendations for management consideration. 
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EXECUTIVE SUMMARY 


Project risk can result in cost overruns, schedule slippage, and performance degradation. 
The capability to assess the current health of a project within a large organization can 
help predict and reduce project failure. A literature survey pointed to INCOSE’s System 
Engineering Leading Indicator (SELI) Guide v2, which provides methods to expose a 
project’s current and potential future states. Displaying trends to reveal leading 
indicators can help facilitate leadership’s necessary actions from leadership to assist a 
project to get on the track for success. This guide emphasizes eighteen SELIs that can 
help leadership identify project risk and to intervene in a situation where a project is 
struggling. All of these SELIs are important; however the scope of this Capstone project 
required selecting a few appropriate SELIs to implement for effective intervention. 

Data elements from projects need to be gathered to produce SELI trends. SSC-A 
currently executes numerous manual data calls to obtain information about 
projects. Coordination and continuous monitoring is involved with the manual data call 
process. Leadership has to decide on how to collect the data, who are the key personnel 
to obtain the information, what information is needed, where can the progress of data 
calls be viewed (e.g., share drive, command website, email, or during scheduled 
meetings), when should the data call deadline be, and why the data call is even 
needed. Skill sets and availability of personnel requested to respond to data calls may be 
inconsistent with the type of data desired and the frequency of requesting this 
information. These types of data calls disturb SE / SE management from their daily 
tasks. Inaccurate information delivered from SE / SE management can produce 
unfavorable results such as data not being received by leadership in a timely manner due 
to repeated errors, which require rework. 

To display SELI trends the current manual data call system and three procurement 
options that could be used to implement the SEHV system were considered. The 
procurement options were automated independent Commercial Off The Shelf (COTS) 
solution, automated centralized GOTS system, or an automated hybrid solution 
set. Analyzing the pros and cons with weighted criteria such as data collection, capture 



and analyze required data elements, cost, data storage, and displaying data trends assisted 
with developing a AHP to determine the best solution. The hybrid solution set that 
involved a combination of an in-house developed database / graphic user interface and 
integration of SE tools / applications was the best solution for SEHV. Elsing these tools 
along with automating manual information gathering processes can support SELI 
visualization needs regarding a project’s success. 

Automating manual data call processes would help alleviate disturbing personnel, 
promote data collection consistency, and allow leadership to receive project information 
on a regular basis. To get the most out of data analysis for the decision-maker, 
visualization is needed. The YBMS project at SSC-A proves this concept by having 
visual depictions of technical data. Leveraging this capability along with SSC-A’s Data 
Center / Enterprise Services Environment can support capturing, storing, analyzing, and 
displaying project health trends. 

Modern SE software tools can produce necessary data elements for SELIs that aid 
the visualization of project trends. There are SE / SE management software and 
applications at SSC-A that are used throughout the command; however, enforcing other 
software that can augment SSC-A’s current SE tool box would be required for applying 
SELI methodology. An automated process would include specific data elements being 
captured from these SE / SE management tools, stored to assist historical trend values, 
and analyzed by performing basic computations on the data to produce trending 
displays. These displays can show clear deviations of planned vs. actual values. These 
deviations could indicate risk on a project and even show if it is a repetitive occurrence. 

Simulations replicate real-life scenarios within a virtual environment that assist 
improvement efforts. Manual data calls were simulated on the software ExtendSim to 
demonstrate how inefficient the current process really was. A sequential process, which 
started from leadership all the way down to the lower levels of the workforce and back, 
confirmed a time consuming set of manual tasks. The SEHV approach reduces some of 
these manual tasks by automating activities, and creating a faster / more efficient 
process. The SEHV methodology was also simulated to indicate that the manual process 
produced outdated “stale” data received by leadership. This stale data would be months 
xxii 



old, which at times prove useless. The SEHV system would include SE / SE 
Management personnel entering data in a common set of SE software tools and 
applications. These tools would provide data elements needed for SELI trends to a 
storage repository. An analytical function would take that stored data and compute 
simple arithmetic to produce arrays of numbers to fuel SELI trends over time. Parallel to 
SE / SE management personnel entering data into SE tools, leadership would log into the 
SEHV system to display project trends at their own convenience. Experiments were 
designed to determine the optimal frequency of SE software tool usage. To keep SELI 
trends fresh and up to date, using SE tools as specified in the SEHV process would be 
necessary. 

Functional architectures for the SEHV system were depicted using SPEC 
Innovations Innoslate to produce IDEFO standard functional models. This model 
displayed functions that were decomposed into lower levels. Providing SEHV, 
performing SE processes, providing SE tool functions, maintaining SEHV, and hosting 
SEHV were the high level activities of the IDEFO architecture. This IDEFO illustrates 
each entity’s relationship with one another via the connected data elements. 

An initial investment in the SEHV system at SSC-A would benefit the command 
by reducing excessive data calls relating to SE / SE management, saving money on man¬ 
hours spent during the manual data call process, providing the capability of displaying 
SELI trends, and the resultant ability to see and react to potential project 
risk. Methodology and recommendations within this capstone project lay the foundation 
for the SEHV capability. This capability automates traditional manual data call processes 
by utilizing technology that is available today to save taxpayer dollars along with 
providing a quality product to the warfighter. 
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I. INTRODUCTION 


This introduction provides a project overview that includes a problem statement 
and the current state of the system operation as it applies to the stakeholders. It contains a 
background discussion, which includes information about the literature review, team 
organization, stakeholders, project objectives, and initial research questions. Finally, also 
included is an approach section that captures the project approach, Systems Engineering 
(SE) approach, and SE Process. 

A. PROJECT OVERVIEW 

This project overview will describe the problem statement; as it highlights the 
boundary of the problem scope. Also, included, is a review of the current system 
operational state with respect to the stakeholders to help emphasize the need for an 
improved system solution. 

1. Problem Statement 

The Space and Naval Warfare (SPAWAR) Atlantic’s (SSC-A’s) engineering 
leadership does not have adequate visibility into the Systems Engineering (SE) health of 
its programs necessary to make timely and accurate decisions concerning potential 
program risks and/or possible mitigation strategies. SSC-A currently performs manual 
data calls to obtain information required to assess a project’s current state of systems 
engineering health. SPAWAR leadership also lacks a means to measure the quality of SE 
work for the projects and programs being executed at the command. With an ability to 
access, store, and analyze the right program information, leadership could gain insight 
into potential systems engineering risk areas and be proactive in resolution of issues that 
lead to program failures or reduced capabilities. 

2. SSC-A’s Current State 

Currently, SSC-A’s leadership conducts manual data calls to obtain necessary SE 
and SE Management information. A System Engineering Health and Visualization 
(SEHV) capability is needed to save time, effort, and ultimately costs by capturing the 
1 



required information more efficiently. The information requested trickles down to lower 
echelons and eventually to the system engineer. In addition to the system engineer’s daily 
activities, response to leadership requested data calls is often expected. This takes quite a 
bit of time when there are multiple projects from which to collect data. An individual 
actually has to manually track down the information requested from each project and 
analyze this information to ensure the correct data is delivered to leadership. Figure 1 
displays the SSC-A’s current state (without SEHV), SSC-A’s SEHV Current State, 
highlights the manual data call process as it exists today. 



Figure 1. SSC-A’s SEHV Current State 
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Having too many people in any process can increase the chances of an 
unsuccessful data call. Leadership’s data call requests trigger meetings for data 
collection. The frequency and duration of these meetings can be decreased significantly 
by capturing this data via an automated process. As manual information gathering tactics 
are taking place, the data call request is usually tunneled down to a lower echelon that 
can provide the data requested. This process can take time, especially when higher 
priority / mission critical tasks are being performed by the information holder. Individual 
projects currently have a system engineer who uses tools to perform SE / SE management 
activities. Once the system engineers provide the information to their chain of command, 
there can be positive or negative feedback given to that system engineer regarding the 
data provided. 

The Veteran’s Benefits Management System (VBMS) project is an example 
within the SSC-A organization that uses visualization software to depict graphical images 
of metrics for their team’s use. The VBMS team uses this type of software with a data 
repository within SSC-A’s Data Center / Enterprise Services Environment. This same 
automated methodology can be used to solve the data call dilemma currently being 
experienced at SSC-A. All stakeholders requesting these measures tend to want to see 
some sort of visual trends so this existing system provides an excellent starting point. 

B. BACKGROUND 

This background section discusses historical material that could have been 
impacted differently with the support of the capability that this capstone report presents. 
The evaluation of the literature related to this topic and used to tailor solutions for the 
project stakeholders is reviewed. The team organization as well as stakeholders’ 
identification for this capstone is provided. Finally, the project objectives and the 
identified initial research questions are discussed. 

1. Project Background 

SSC-A was highly in need of a comprehensive understanding, through dynamic 
displays, of the overall health of its SE process areas against prescribed performance 

measures. SSC-A has the need to access specific SELIs to provide decision-makers 
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dynamic and current visibility into quality and effectiveness of SE / SE management 
activities. The capability is expected to assist SSC-A in the identification and diagnosis of 
potential SE / SE management problems either before they occur in order to mitigate 
them or after they occur in order to prevent like problems in the future. 

The data housed in SSC-A’s existing SE toolsets only provide a snapshot of the 
status of the project at the moment in time when the data is requested. SSC-A does not 
have visual trending representations of the status of SE management processes that can 
provide visibility into expected project performance and potential future states. An audit 
trail of data needed to be captured, stored, aggregated, and analyzed from the various 
projects in an automatic or non-disturbing fashion to display the respective trending 
status. The SEHY team proposed various visual representations of the data in order to 
illustrate the measures extracted from projects to provide SEHV stakeholders an 
indication of potential future project states. 

SSC-A strives to offer continuous process improvement when providing quality 
systems to stakeholders. The command’s goal is to be more proactive identifying system 
health indicators during early phases of system development. The savings from early 
identification and monitoring of system health could have been realized in the following 
examples if an SEHV capability had been present. The Defense Information Management 
Human Resource System (DIMHRS) is a failed project that did not have proper insight 
into SE health issues. Almost 1 billion dollars was spent on DIMHRS and still the project 
failed to deliver the capability promised. The deliverables were supposed to be used 
across all Department of Defense (DOD) branches of service but ended up only being 
used by the Army and Air Force. The lack of written requirements and configuration 
management control were responsible for the issues with DIMHRS. Leadership of 
DIMHRS directed engineers, on several occasions, to develop the project and forego the 
documentation until later. Each branch of service wanted it tailored to suit its respective 
needs. This fueled unrealistic schedules and continuous requirement changes. The 
resulting outcome of the DIMHRS project came nowhere near the overall expectation 
(Ugone 2010). 
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Had System Engineering Leading Indicators (SELIs) been used on the DIMHRS 
project, specifically using requirements trends (SELI 1), the failure outcome could have 
been projected prior to realization. As stated in the previous paragraph, “the lack of 
written requirements” was one of the biggest issues that were identified as responsible for 
project failure. Had DIMHRS used requirements trends and kept track of them, the 
resulting outcome may have been successful. Because that there were no requirements 
trends being tracked, there was no way to test validation (SELI 4) and verification (SELI 
5) of the project. Furthermore, using Risk Exposure trends (SELI 9), Risk Treatment 
trends (SELI 10), and Process Compliance Trends (SELI 12) might have helped forecast 
the risks and identify options to correcting them. The Process Compliance trends would 
have also aided in maintaining configuration management. Had the SEHV process been 
applied to the DIMHRS program, as stated, the program might have been completed on 
time and under budget. 

SSC-A has also developed human resource systems, such as the Navy Standard 
Integrated Personnel System (NSIPS), the Career Information Management System 
(CIMS), and the manpower control system, Total Force and Fleet Manpower System 
(TFFMPS), all of which span the complete software development life cycle from 
conception to sustainment. SSC-A developed the Veteran Affairs’ (VA) Veterans 
Relationship Management/Customer Relationship Model (VRM/CRM) and a Master 
Veteran Index (MVI) system and has integrated these systems with numerous other 
legacy VA systems. There are several other projects that are in progress that include 
concept refinement, development, integration, and potential sustainment of software that 
are within different stages of the project life-cycle that could truly benefit from this type 
of capability. The SEHV solution provides the ability for each individual project to 
explain its own unique representative systems engineering status in order for automated 
analysis of project health to be identified and visualized to command leadership. 

Most of these programs, if not all, were rated as semi-successful to successful 
projects. However, NSIPS is highly modified using custom coding rather than using 
COTS software by PeopleSoft, now Oracle code. Had all these programs with NSIPS 
used Interface Trends (SELI 3), Risk Exposure Trends (SELI 9), Risk Treatment Trends 
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(SELI 10) and Architecture Trends (SELI 17), they may have been more successful than 
they are today. By using SELI 3 and SELI 17, there would have been a better architecture 
in place and fewer problems when Oracle updated its systems including less interface 
problems with each new version. The use of SELI 9 and SELI 10 could have exposed 
risks and possibly provided insight on what was necessary to mitigate them properly. The 
same applies to CIMS since it is part of NSIPS. 

TFFMPS might not have been shelved had it utilized the risk and interface trends. 
Following the guidance of SELI 3, SELI 9, and SELI 10, TFFMPS may have been in 
production today. The biggest issue found with using TFFMPS is that it cannot interface 
with an excel document into an Oracle/PeopleSoft COTS product. A fit gap analysis of 
the excel spreadsheet to use in the COTS product should have been conducted to make 
this determination early in the life cycle. 

VRM/CRM and MYI used a modified Agile system and followed it as planned, a 
modified Agile system is defined as a combination of a Waterfall and an Agile 
development plan using good engineering practices. Certainly, the architecture could 
have used some modifications. The architecture could have been improved had it used 
SELI 3 and SELI 17. If SELI 3 and 17 had been considered when relating legacy systems 
to the new system, VRM/CRM and MVI risk exposure and treatment plan would not 
have been so extensive to overcome most of the interface issues. 

2, Literature Review 

A literature review was conducted and eventually prompted the use of 
International Council on Systems Engineering (INCOSE) SELIs Guide version 2.0 as the 
major guidance for the capstone effort. The team’s initial research consisted of exploring 
and reviewing different publications, briefs, and artifacts related to measurement of SE 
management processes. The team’s initial review provided insight on how SE metrics are 
currently being utilized and what information is being revealed. 
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MIT previously conducted research to determine how to use SE metrics to 
identify problem areas and to predict trends. The study breaks down characteristics of 
what constitutes a leading indicator and its specific usefulness. Objectives and goals were 
also contained within the document (Rhodes, Yalerdi and Roedler 2008). 

Paul Montgomery and Ron Carlson provided, in an NPS literary work on April 
30, 2010, a discussion about the Applied Leading Indicators (ALI) SE tool. Data analysis 
features were illustrated that Naval Air (NAVAIR) used for their SE process areas. 
Historical metrics were obtained and analyzed to support the goal of predicting future 
performance (Montgomery and Carlson 2010). 

INCOSE’s SELI Guide version 2.0 technical product identified leading indicators 
that were associated with various parts of the project life cycle and include work from 
reputable sources such as the Massachusetts Institute of Technology (MIT), Lockheed 
Martin Corporation, INCOSE, the Lean Aerospace Initiative, the Systems Engineering 
Advancement Research Initiative, as well as Practical Software and Systems 
Measurements (PSM). Trends and various measurements were defined within the 
technical product to provide examples of needed metrics. The ALI tool was briefly 
discussed and its capabilities/limitations were clearly defined (INCOSE et al. 2007), 
(INCOSE etal. 2010). 

3. Team Organization 

Figure 2 provides the high-level organizational structure chart for the SEHV 
Capstone Project. 
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Figure 2. SEHV Capstone Organizational Structure 


The SEHV team was organized according to key activities necessary for 
completion of the project. Each activity had a primary and alternate team member 
assigned. The key activities for this project were project planning, requirements 
development and validation, mapping of data elements to tools, SELI metrics analysis, 
cost estimation, process modeling, risk assessment, identification of use case scenarios, 
and an analysis of proposed metrics and process sets. The capstone advisors reviewed the 
scope, organization, quality, and schedule of the project and provided feedback on a 
regular basis. The team engaged with the stakeholders to identify use cases, requirements, 
and their priorities. Table 1 lists the SEHV team roles and responsibilities and identifies 
each team member(s) assigned. 




Table 1. SEHV Capstone Project Team Roles and Responsibilities 


Team Member 

Roles and Responsibilities 

Project Manager and Project 
Technical Risk Analyst: 

Primary-Clive Sugama 
Altemate-Chester Alonzo 

Responsible for planning, monitoring, and controlling 
activities of the project. Lead development of the PMP. 
Responsible for Project / Technical Risk Analysis and 
Configuration Management (CM). Assisted with 
identifying key stakeholders and facilitated discussions 
with stakeholders for the purpose of capturing 
requirements, use cases, and metrics priorities. 

Lead Systems Engineer (LSE): 

Primary-Chester Alonzo 
Altemate-Clive Sugama 

Lead the elicitation, documentation, and organization of 
the stakeholder requirements. Captured system use 
cases from the various stakeholders. Tracked and 
managed the interfaces for the architecture. Tracked and 
managed the activities performing the verification and 
validation of the requirements, design, and architecture. 

Data Architecture Lead: 

Primary-Michael Besco 
Altemate-Theresa Inman 

Lead the development of the architecture for the 
system. Lead in identifying leading indicators that 
would indicate the health of system engineering efforts 
within SSC-A. Additionally, developed the entity- 
relationship architecture from the SE tools that will map 
data to the leading indicators. 

Software Analyst: 

Primary-Theresa Inman 
Altemate-Michael Besco 

Analyzed the implementation of the architecture. 
Interfaced with the Architecture Lead from SSC-A 
Enterprise SE Tool Suite group to determine the data 
elements that were harvested from the selected tool 
suites and recommended a data strategy for the 
automated collection, storage, and archival of the data. 

Data Editor: 

Primary-Regina McNeil 

Alternate- Michael Jourdain 

Ensured all documentation was formatted, consistent, 
and contained the information pertinent to that 
deliverable. Enforced the standards set by the NPS 
Thesis Processing Office on this SEHV project report. 

Cost Estimator: 

Primary-Michael Jourdain 
Altemate-Regina McNeil 

Provided cost and benefit analysis on the SEHV 
capability recommendation. Analyzed costs associated 
with implementation of metric data collection and 
provided cost projections associated with SEHV 
development. 




4. 


SSC-A Active Stakeholders 


Figure 3 displays the Operational View (OV-4) of Active Stakeholders which 
identifies roles within the SSC-A organization that were involved in the use of the SEHV 
System. 
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Figure 3. OV-4 SSC-A Active Stakeholders 


5. Project Objectives 

The objectives of this Capstone Project were to develop the requirements, 
architecture, data strategy, and conceptual model for the SEHV capability. The SEHV 
capability context diagram in Figure 4 illustrates the proposed SEHV capability and 
helped the team to bond the project objectives and scope. Projects were the source of the 
data elements for both the SE / SE management data elements. SE / SE management tools 
or other sources were used to generate these data elements. The SEHV capability that is 
housed within the Data Center / Enterprise Services Environment (DC/ESE) used the 
guidelines within the INCOSE SELI Guide version 2.0 to depict project trending data to 
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SEHY stakeholders. The stakeholders were able to review these SELI trends using data 
collected from projects and provide feedback. 



Figure 5 displays the OV-1 which illustrates the SEHV capability. The system 
engineer and project team allocated to a specific project produced data using SE / SE 
management tools. The data elements from these tools and other sources were harvested 
or collected by non-disruptive means and kept in the data repository located within the 
DC/ESE at SSC-A. The VBMS project has a Visual Depiction Capability and Data 
Repository within the DC/ESE which was leveraged for the SEHV capability. This 
capstone project researched means to utilize these capabilities to the fullest when 
introducing the SEHV capability. SELI analytics will be a core piece of the visual 
depiction capability. The visual depiction capability provides displays of SE health to 
SEHV stakeholders in order to initiate and facilitate feedback to the system engineer of a 
project. 
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Feedback 



Figure 5. SEHV Capability OV-1 


Key SE health metrics will provide insight into the quality and effectiveness of 
SE activities which support command objectives of consistent solution delivery and 
customer satisfaction. Functional and physical architectures will support a depiction of 
the collection, evaluation, and dissemination of relevant dynamic data elements that 
supported the validated SELI measures. The architecture will also support temporal and 
pattern analysis of the data elements. A method to ingest the identified data elements into 
the SEHV data repository, in an automated manner, will be included. 

6. Research Questions 

Working with the stakeholders’ requirements and current state of technology, the 
SEHV team developed system engineering research questions that guided the research 
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and the system conceptual design. These questions formed the basis for understanding 
stakeholders’ needs and system goal of this project. The answers can be found in 
Appendix D. 

Data Sources 

• Which tools and supporting processes capture the data elements needed to 
ingest into the SEHV solution? 

• What engineering tools are being leveraged? 

• What data sources are available? 

• What data elements are needed? 

• What tool data is most dependable or valuable? 


Data Collection 

• How is data currently being captured? 

• How will the automatic collection work? 

• If the data cannot be collected automatically, can the SELI data elements 
be collected on some sort of schedule? 

• How easily can one collect any specific piece of data? 


Data Storage 

• How are these data elements (individual pieces of data) related to one 
another? 

• What questions will be asked of the data? 

• How long will the historical data need to be stored? 

Data Analysis 

• Can a process be constructed to avoid data calls? 

• What calculations need to be performed on the data elements? 

• What trending patterns are being looked for in the data? 
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Data Display 

• What visualizations best represent the data? 

SEHY Stakeholders 

• Who are the stakeholders that need to view these SELIs? 


SELIs 

• What are the top SELIs that will provide the most insight to leadership for 
determining how well the SE project is going? 

• Which SELIs do the stakeholders find most-useful? 


Other Topics 

• What is good systems’ engineering / management? Is it similar to good 
project management with ensuring the SEHV system functions satisfy 
stakeholder requirements? 

• How are other similar commands monitoring SEHV? (Ex. SSC-PAC, 
NavFac, NavAir, Army Corps., AF ESC) 

• How much will it cost to collect SELI data elements? 

• How long should it take to obtain a required visualization? 

• How many metrics or trends should be viewed? 


C. APPROACH 

Due to the complexity of the stakeholders’ needs and current state of technology, 
the SEHV team agreed on the encompassed project approach and system engineering 
process for the development of the proposed SEHV system. 

1. SEHV Project Approach 

The SEHV team analyzed SE leading indicators to facilitate project risk 
identification. The team studied the INCOSE SELI Guide version 2.0. This guide was 
used as a control for this project. This guide identified 18 leading indicators that provided 

consistent insight into future project performance. The use of the SELIs ensured levels of 
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quality for the solution. SELIs were reviewed and assigned priorities based on their 
impact to project cost and schedule, and system performance as well as their ability to 
answer common questions by engineering managers concerning system risks. Table 2 
lists the 18 SELI trends documented in the INCOSE SELI Guide version 2.0. 


Table 2. SELI Trends 


SELI 1. Requirements Trends 

SELI 2. System Definition Change Backlog Trends 

SELI 3. Interface Trends 

SELI 4. Requirements Validation Trends 

SELI 5. Requirements Verification Trends 

SELI 6. Work Product Approval Trends 

SELI 7. Review Action Closure Trends 

SELI 8. Technology Maturity Closure Trends 

SELI 9. Risk Exposure Trends 

SELI 10. Risk Treatment Trends 

SELI 11. Systems Engineering Staffing and Skills Trends 

SELI 12. Process Compliance Trends 

SELI 13. Technical Measurement Trends 

SELI 14. Facility and Equipment Availability 

SELI 15. Defect and Error Trends 

SELI 16. System Affordability Trends 

SELI 17. Architecture Trends 

SELI 18. Schedule & Cost Pressure 


The SEHV team’s project approach to uniquely defining and applying the 18 
SELIs was based on criteria such as the interest in the SELI by multiple stakeholders, 
ease of automating the method of capturing the data elements that support the SELI, tool 
availability that would produce data elements that support the SELI, having existing and 
available SE / SE management tools, and the cost of tools. The data collection process of 
the SELI choices documented how much an automated solution that harvests the SELI 
measures would be able to reduce expended time. Once data was collected, data 
visualization options were researched to be able to display trends of SELI measures. 
Figure 6 shows the SEHV project approach which includes SELI choices, data collection 
process, and display mechanism options which result in the final SEHV reco mm endation. 
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Final SEHV Recommendation 

Figure 6. SEHV Project Approach 

2. SEHV SE Approach 

The SEHV team utilized an SE approach derived from the IEEE 1220 2005 
Systems Engineering Process. This process was appropriate for this project since the 
approach produced documents and models versus hardware or software. Using this 
process helped facilitate the project’s expected process environment, interfaces, work 
products, and necessary SE tools needed for the SEHV capability. This process began by 
obtaining SSC-A stakeholder inputs, then going through the SEHV SE process, and 
ultimately concluding with a recommendation (based on weighted criterion) of SELI 
choices that described data measure collection methodologies and their respective display 
mechanism s . 

The SE approach input included evaluating SSC-A’s requirements for an 
automated solution that can display project’s SE trends that utilize leading indicators to 

suggest potential project risk. Evaluation of design requirements took place to include 
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identifying key SE / SE management data elements used to produce SE trends. Functional 
and nonfunctional requirements were also decomposed and validated to ensure the SEHV 
capability was within the scope of the NPS capstone. 

The analysis approach began with defining the problem of SSC-A’s limitation of 
not being able to determine a project’s SE status in a timely manner. A literature review 
resulted in multiple sources pointing to INCOSE’s SELI Guide version 2.0. It describes 
specific data elements needed in order to produce SE trends that expose potential risk 
areas. To determine the appropriate criteria for SSC-A, the Competency Aligned 
Organization (CAO) Concept of Operations (CONOPS) and Integrated Product Team 
(IPT) handbook were used to evaluate which SELI would be most important to a 
particular role within the organization. Evaluating techniques such as selecting most 
valued SELI and defining modeling requirements contributed to construction of 
Integration Definition 0 (IDEFO) architectures for particular SELIs. The use of actual 
data was not able to be obtained from sources such as Navy Enterprise Resource Planning 
(NERP), but expected data elements were used to produce examples of SE trends over 
time. A simulation was created using input variations to display a plethora of results for 
analysis. Risks, assumptions, and other applicable criteria were analyzed based on the 
capstone project’s findings, and recommendations for the SEHV solution options were 
provided based on analysis results. 

An overall analysis of the SEHV solution including the necessary features and 
requirements needed for the SSC-A organization to produce this needed capability was 
performed. Recommendations which result in decisions to move forward with the SEHV 
solution were documented to facilitate appropriate action for funding a prototype for this 
effort. The resulting capstone output includes this report that serves as a firm foundation 
for the SEHV capability. 

3. SEHV SE Process 

Requirements analysis and validation included needs analysis, stakeholder 
analysis, SELI prioritization, data source identification and an overall requirements 
review. Needs analysis included determining the need of SSC-A stakeholders of having 
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consistent command-wide visibility of projects to determine potential project risks. 
Stakeholder analysis was used for organizational structure which a OV-4 was developed. 
SELI prioritization was facilitated by SSC-A authoritative documentation such as the 
CAO CONOPS and IPT Handbook to satisfy schedule requirements of the SEHV 
Capstone. Data source identification showed required data elements for SELI 
visualization trends. 

Functional analysis and validation included defining data collection, identifying 
supporting processes, completing IDEFO modeling and a design review. Defining the 
data collection was pulling the required data elements to support SELI trends from SE 
software tools. SE / SE management would routinely input data within the tools to 
provide up to date data collection by the SEHY system. Identifying supporting processes 
was required for the SEHV methodology to use features such as capture, store, analyze, 
and display SELI trends. IDEFO modeling was performed using SPEC Innovations 
Innoslate to illustrate a functional architecture of the SEHV system with respect to the 
chosen SELIs. Entity Relationship Diagram (ERD) modeling would represent 
relationships within the SEHV system to understand interaction between neighboring 
components. A use case analysis was planned for the design verification phase to 
simulate the SEHV system in action. 

Design verification consisted of creating an experimental architecture, modeling 
historical project data, performing statistical analysis, and proposed visualizations / 
graphs. Creating an experimental architecture was done by Imagine That Inc.’s 
ExtendSim to simulate a manual data call process along with the proposed SEHV 
process. This simulation would prove the SEHV process is more efficient than the current 
manual data call process. Modeling historical project data would be necessary by 
obtaining data elements from SE tools over a period of time in order to show visual 
trending information. A repository to keep historical project data would be required to 
depict positive or negative SELI trends over time. Performing statistical analysis was 
done by SAS Institute Inc.’s JMP software to determine the frequency of SE tool usage 
within the SEHV process. A minimum amount of SE tool usage was determined to 
provide fresh data. Proposing visualizations and graphs would include notional examples 
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where possible displays could be used for the SSC-A SEHV initiative. Figure 7 displays 
the SEHV SE process. 



Figure 7. SEHV SE Process Diagram 


The five functional blocks in Figure 8 represent a high level grouping of the 
activities outlined in the SE process in Figure 7. These SE processes were derived and 
tailored from the IEEE 1220-2005 Systems Engineering Plan (SEP). It also categorizes 
when each of the high level functions occurred within the three quarters allocated to this 
capstone project. Figure 8 displays a description of activities performed for Problem 
Definition/Needs Analysis, Requirements Analysis/Metrics Analysis, Functional 
Analysis, System Architectural Modeling, and System Analysis. 
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Figure 8. SEHV High Level Project Plan 


a. Problem Definition/Needs Analysis 

The team defined the problem and performed a needs analysis for the SEHV 
capability. This included conducting the stakeholder needs analysis, completing an 
analysis of SSC-A current capabilities, and providing examples of programs that have 
failed or had issues as a result of a lack of insight into their SE health. Analysis of the 
SEHV capability needs determined the overall boundaries, limitations, and constraints. 

SSC-A needed dynamic displays to illustrate the overall health of its SE process 
areas against prescribed performance measures. SSC-A has a need to assess specific 
SELIs to provide decision-makers dynamic and current visibility into the quality and 
effectiveness of its SE/SE management activities. The capability assisted SSC-A in the 
identification and diagnosis of potential SE / SE management problems either before they 
occurred to mitigate them or after they occurred in order to prevent like problems in the 
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future. The entry criteria included SSC-A stakeholders inputs; and the exit criteria 
included documented stakeholder needs. 

b. Requirements Analysis/Metrics Analysis 

The team performed requirements analysis that resulted in a functional 
requirements baseline for the solution. Requirement analysis began with ensuring that the 
SSC-A stakeholder’s needs were adequately documented, understood, and suitable for 
refinement. These were reviewed against the project criteria, and agreed upon by the 
SSC-A active stakeholders, NPS advisors, and the SEHV Capstone Project team. The 
SEHV team analyzed SEHV leading indicators to facilitate project risk identification. 
The team studied the INCOSE SELIs Guide version 2.0, used as a reference for this 
project. The INCOSE SELI guide identified 18 leading indicators that have been proven 
to provide insight into future project performance. The use of the SELIs ensured levels of 
quality for the solution. SELIs were reviewed and prioritized based on their impact to 
project cost and schedule, system performance, as well as, on their ability to answer 
common questions posed by engineering managers concerning system risks. The entry 
criteria included the project need statement; and the exit criteria included a stakeholder 
analysis, documented SEHV solution functional requirements, and a prioritized listing of 
SELIs based upon SSC-A stakeholder needs. 

c. Functional Analysis 

The automated SEHV solution will consist of functions such as capture, store, 
analyze and display. These functions will perform in an environment where other existing 
SSC-A tools will be present. The tools such as NERP, Total Workforce Management 
System (TWMS), Demand Signal Tool, and Risk Exchange will help nurture the SEHV 
system with the information it needs. Capturing data from these tools would involve a 
request from the SEHV user interface requesting the desired data. This information will 
be stored along with the users’ input into a data repository. The SEHV analyzing 
capability would perform basic arithmetic to sum up the values of the data. In most cases 
this is a comparison of user planned information versus actual values provided by SE / 
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SE management tools. The SEHV analyzing capability would also set arrays of data for 
the display function to manipulate the data into a visual depiction. 

The SEHV team developed an IDEFO diagram and an ExtendSim simulation as 
part of the functional analysis. The entry criteria of the functional analysis section 
included the data elements, data sources, and notional trending patterns of the most 
valued SELIs. The exit criteria included an IDEFO diagram and ExtendSim model 
depicting data capture methods, user input triggers, storage strategies, data analyzing 
functions, and visualization features. 

d. System Architectural Modeling 

The team continued to refine the SEHV context diagram, as well as, the SSC 
Current State and SEHV Capability OV-1 diagrams. The team developed an architecture 
model to be used to facilitate development of the SEHV capability. Functional and 
physical architecture models were created to assist in implementing the SEHV capability. 
The functional architecture model is solution neutral while the physical architecture 
model contains components that were evaluated against specific criteria such as cost, 
availability, and feasibility. The team developed an ERD to define the physical data 
model of the prospective relational database structure. The entry criterion included SEHV 
IDEFO diagrams, positive, negative, and neutral SELI trending patterns, and data 
elements with associated source information. The exit criteria included a functional 
architecture that allocated SEHV functions to specific components, as well as, an ERD 
that facilitated the storage and analysis of the SEHV dataset. 

e. System Analysis 

The team performed an assessment of the SEHV capability against the functional 
requirements and operational use cases. Historical project data was mapped to the 
proposed entity relationship model and the team analyzed the SEHV capability in 
identifying trends and insight into SE health. Notional graphs and dashboards were 
created to present the information captured within the SEHV solution to the stakeholders. 
The team performed a high-level cost analysis of implementing the system and a risk 
analysis to identify any risks that SSC-A may encounter in implementing this SEHV 
22 



capability. The entry criteria included the functional architecture and physical 
architecture and the exit criteria included a validated architecture against historical data 
and operational use cases along with potential visualization mechanism s . 

4. Risk Management 

A distinction was made between overall project risk management and IT system 
or application risk management. The project manager worked with the project team and 
project sponsors to ensure that risks were actively identified, analyzed, and managed 
throughout the life of the project. Risks were identified as early as possible in the project 
to minimize their impact. The steps for accomplishing this are outlined in the following 
sections. The PM or Risk Assessment Manager served as the Risk Manager for this 
project. The steps for accomplishing this are outlined in Appendix A. 
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II. REQUIREMENTS ANALYSIS 


Perhaps one of the most challenging efforts to be realized during this project was 
the requirements analysis process. After understanding stakeholder needs, the team 
elicited requirements to be analyzed and organized them by high level function. The 
decomposed stakeholder requirements are shown in the following sections. Next, a 
functional analysis using modeling software to connect required functional abilities 
which helped verify SELI interactions was performed. In the final section, due to the 
realization that performing full detail analysis for all SELIs would become a schedule 
risk for this effort, a down select of five commonly discussed indicators, at SSC-A, was 
performed. 

A. REQUIREMENTS ELICITATION 

The SEHV team elicited requirements from the SSC-A Stakeholders. The 
stakeholders articulated their high level requirements, which are listed in section 1. From 
these high level requirements, the SEHY team decomposed the high level requirements 
into the component functions that would be required for the system. Additionally, the 
SEHV team identified supporting functions that are required for the ongoing operations 
and sustainment of the solution, as well as, for the expected operating environment 
requirements. 

The SEHV team performed functional analysis and allocation of the requirements 
within Innoslate version 3.0. Each requirement was documented and managed within 
Innoslate. The established requirements were analyzed by the team to ensure they were 
clear, concise, measurable, and testable. The full list of the system requirements baseline 
is documented in section 2 below. 

1. Stakeholder Requirements 

The SEHV team worked with SSC-A Stakeholders to produce the following high 
level requirements for the SEHV system. 

• The system shall perform automated data capture 
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• The system shall use data elements that can be pulled from SE tools and 
other data sources 

• The system shall provide leading indicator analysis 

• The system shall display SELI trends 

• The system shall apply SELI concepts 

2. SEHV Decomposed Requirements 

Decomposition was performed by further research and evaluation of industry best 
practice in the SE community as well using a specific application for SSC-A 
organizational functions. Requirements decomposition established one of the essential 
requirements traceability relationships. It shows how each layer of requirements 
contributed to the satisfaction of the layer above. It is this relationship that connected the 
conceptual design to the development of requirements (IBM 2013). 

The below list represents the SSC-A stakeholder validated decomposed functional 
and supporting requirements applicable to the SEHV system. 

SEHV Functional Requirements 

1. Data Capture 

The system shall perform automated data capture 

1.1. Data Pull 

The system shall pull data from existing SE tool data sources 

1.2. Data Filter 

The system shall filter results pulled from interfacing data sources 

2. Data Storage 

The system shall store captured data from SE tools databases 

2.1 Data NormalizationThe system shall transform data from external 
data sources into a standard format for storage 

2.1.1 Interfaces 

The system shall receive data from external data sources. 

2.1.1.1 Data Source: IBM DOORS 

The system shall receive data from IBM 
DOORS 
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2.1.1.2 Data Source: Risk Exchange 

The system shall receive data from Risk 
Exchange 

2.1.1.3 Data Source: JIRA 

The system shall receive data from JIRA 

2.1.1.4 Data Source: NERP 

The system shall receive data from NERP 

2.1.1.5 Data Source: TWMS 

The system shall receive data from TWMS 

2.1.1.6 Data Source: TAA 

The system shall receive data from TAA 

2.2 Data Commit 

The system shall store data in the SEHV database 

2.3 Manage Data Requests 

The system shall receive data from external data sources at 
configurable frequencies 

2.4 Transaction Logging 

The system shall log database transactions 

2.5 Database Search 

The system shall provide a database search capability 

2.6 Database Indexing 

The system shall provide database indexing services 

2.7 Database Backup 

The system shall provide database backup services 

2.8 Database Archival 

The system shall provide database archival services 

2.9 Database Recovery 

The system shall provide database recovery services 

2.10 Database Auditing 

The system shall provide database auditing services 
Data Analysis 

The system shall perform analysis on the collected data 



3.1 Perform Calculations 

The system shall perform calculations on the captured data 

3.2 Pattern Analysis 

The system shall have the ability to identify specific patterns in the 
collected data 

3.3 Statistical Analysis 

The system shall have the ability to perform statistical analysis on 
the collected data 

3.4 Time Based Analysis 

The system shall have the ability to perform temporal analysis on 
the collected data 

4. Data Display 

The system shall display SELI trending graphs 

4.1 Configurable Dashboard 

The system shall allow the user to select a graph to view 

4.1.1 SELI Graphs List 

The system shall list SELI trending graphs that are 
available for use 

4.1.2 Proj ect Listing 

The system shall allow the user to select from a list of 
projects, which are organized by portfolios and IPTs, to 
display on the graph 

4.1.3 User Profile Settings 

The system shall save user profile settings such as graph, 
project selection, and dashboard configuration 

4.2 Provide SELI Visualizations 

The system shall display SELI trending graphs. 

4.2.1 System Affordability Trends 

The system shall display System Affordability trending 
graphs 

4.2.2 Process Compliance Trends 

The system shall display Process Compliance trending 
graphs 

4.2.3 Risk Exposure Trends 

The system shall provide Risk Exposure trending graphs 
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4.2.4 Technical Measurement Trends 

The system shall display Technical Measurement trending 
graphs 

4.2.5 SE Staffing and Skills Trends 

The system shall display SE Staffing and Skills trending 
graphs 

4.2.6 Additional SELI Trends 

The system shall be capable of handling up to 13 additional 
future SELIs 

4.3 Administer SEHY Dashboard 

The system shall allow privileged users to adjust exposed 
configuration settings from a web interface 

4.3.1 Manage Graphs 

The system shall allow a privileged user to add, edit, or 
remove the list of graphs available for use 

4.3.2 Manage Portfolio/IPT/Project List 

The system shall allow a privileged user to manage the 
hierarchical relationship of Portfolios/IPTs/and Projects 

4.3.3 Manage Prediction Calculators 

The system shall allow a privileged user to manage the 
calculations used within the graphs 

4.3.4 Manage Users 

The system shall allow a privileged user to manage the list 
of users and their roles 

5. Supporting Infrastructure 

The system shall provide the hardware and software infrastructure to 
support the solution 

5.1 Web Application Hosting 

The system shall provide web application hosting services 

5.1.1 Concurrent Users 

The system shall provide access to number of concurrent 
users which will have to be determined in the future 

5.2 User Management 

The system shall provide management of user accounts 

5.2.1 Role-Based Access 
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The system shall implement role based access security 

5.2.2 Authentication Mechanism 

The system shall provide a user authentication mechanism 
which complies with all applicable information security 
and command security requirements 

5.2.3 Restricted Access 

The system shall prevent access of SEHV data by 
unauthorized users 

B. FUNCTIONAL ANALYSIS 

Functional analysis was performed with a Model-Based Systems Engineering 
(MBSE) approach. The functional model was created in Innoslate as an IDEFO model. 
Innoslate assisted with the analysis of the model by providing metrics on its maturity by 
analyzing the decomposition of the functions; analyzing the requirements traceability and 
determining whether the functions had triggers or controls in place. 

The five top-level functions are listed below. 

1. Provide SEHV 

The “Provide SEHV” function denotes the top-level function of the main 
application. “Provide SEHV” is decomposed by “Capture Data,” “Store Data,” “Analyze 
Data,” and “Display Data” functions. The “Capture Data” function interfaces with 
external data sources. The “Store Data” function encapsulates all the component 
functions associated with data storage. The “Analyze Data” function decouples the logic 
layer from the data tier and allows the solution to be easier to maintain and upgrade 
component parts. The “Display Data” function includes the features that allow the user to 
access and interact with the SEHV system. 

2. Perform SE Activities 

The top-level “Perform SE Activities” function incorporates all of the associated 
processes that are external but related to the SEHV system. “Perform SE Activities” is 
broken down into four functions, which are “Request System Engineering Services,” 
“Perform System Engineering Services,” “Use SEHV to Assess System Engineering 
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Services,” and lastly “Provide SE Process Guidance.” External customers request 
systems engineering services from SSC-A via the “Request System Engineering 
Services” function. The activity of a systems engineer at SPAWAR working on a project 
is represented by the “Perform Systems Engineering Services” function. The “Use SEHV 
to Assess Systems Engineering Services” function is where 5.0 leadership at SSC-A 
analyzes the graphs produced by the SEHV system to evaluate the quality of systems 
engineering being performed. Armed with the trending data from SEHV for projects as 
SSC-A, 5.0 leadership now has sufficient data to inform decisions, such as, where to 
apply resources, determine where gaps in training are, or advertise to potential customers 
what SSC-A strengths are in order to get more systems engineering work in that area. 
Finally, the “Provide SE Process Guidance” functions represent the resources available to 
a systems engineer at SSC-A. 

3. Provide SE Tools Functions 

The “Provide SE Tools” function groups all of the SE toolsets that must interface 
with SEHV. The tools are listed by functions within “Provide SE Tools” in order to 
provide a level of abstraction. The sub functions of “Provide SE Tools” are “Provide 
Requirements Management Tool Functions,” “Provide Risk Management Tool 
Functions,” “Provide Project Staffing Tool Functions,” “Provide Process Compliance 
Tool Functions,” “Provide Project Costing Tool Functions,” “Provide Personnel Data 
Tracking Tool Functions” and “Provide Demand Signal Tracking Tool Functions.” 

4. Maintain SEHV 

The “Maintain SEHV” function consolidates all of the functions that contribute to 
the maintainability of SEHV. The “Manage System Updates” subfunction tracks all 
corrective and preventative updates to the SEHV system. The “Manage System 
Configuration” subfunction ensures that configuration details of the SEHV system are 
easily managed and configurable. The “Manage System Interfaces” subfunction tracks 
the interface details for each external data source that interfaces with SEHV. The systems 
interfaces must also be easily managed and configurable. The “Manage Data Capture 
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Requests” subfunction manages the frequency that data is harvested from external data 
sources. 

5. Host SEHV 

The “Host SEHV” top-level function identifies the component functions that are 
required to host SEHV. These are the hosting services that would be required in order to 
host SEHV within the data center at SSC-A. The “Manage Database” subfunction is 
performed by a database management system. The database management system 
provides capability such as data indexing, data archive, data backup, and data restore. 
The “Manage Web Application” function is performed by a web application server. The 
web application server provides the capability to serve out the web application to remote 
users. The “Manage Users” function is performed by a user account management tool, 
such as Active Directory, and provides the capability to authenticate users to access the 
SEHV system. The “Manage Storage” function is performed by the storage volume that 
handles the storage of the SEHV data. The storage volume provides the capability to 
allocate adequate storage for SEHV as well as provide failover and redundancy in order 
to increase the reliability and performance of SEHV. 

All associated IDEFO models are provided as screen captures on a supplemental 

disc. 


C. SELECTION OF LEADING INDICATORS 

A leading indicator is a measure for evaluating the effectiveness of how a specific 
activity is applied on a project to provide information about impacts that are likely to 
affect system performance objectives. A leading indicator may be an individual measure, 
or collection of measures and associated analysis predictive of future systems engineering 
performance before the system is fully realized. Leading indicators aid leadership in 
delivering value to customers and end users, while assisting in taking interventions and 
actions to avoid rework and wasted effort. By analyzing the trends, predictions can be 
forecast the outcomes of certain activities. Trends are analyzed for insight into both the 
entity being measured and potential impacts to other entities. This provides leaders with 
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the data they need to make informed decisions and where necessary, take preventative or 
corrective action during the program in a proactive manner. 

1. SELI Down Select Process 

The SEHY team performed an SELI down select process to choose a subset of 
SELIs to analyze in depth. In order to comply with the capstone schedule requirements, 
the team selected five of the 18 SELIs to study. The team’s intent was to study a subset of 
the SELIs that were of importance to the SSC-A organization and that would be 
representative examples whose results would be extensible to the full set. The subset of 
SELIs was chosen to lay the foundation of the SEHV methodology. All SELIs were 
considered important as indicators of SE health; however to successfully complete the 
Capstone report within the allotted time, the down select process was necessary to focus 
on a specific set of SELIs. The down select process allowed selection of SELIs based off 
of authoritative SSC-A documents which mapped organizational roles to these SELIs. 

Mapping to Interested Stakeholders 

SSC-A’s organization involves various stakeholders with specific roles and 
responsibilities. The SSC-A Competency Aligned Organization (CAO) CONOPS and 
IPT Handbook released a new version of documentation in March 2014 which 
characterizes the organization’s personnel. These documents have been advertised as the 
command’s authoritative source of information. Descriptions of stakeholders were 
closely matched to the 18 SELIs listed in INCOSE’s SELI Guide version 2.0. Thi s 
enabled the team to determine which stakeholders were assigned responsibilities that 
incorporated risks depicted in INCOSE’s SELI Guide version 2.0 to support down 
selecting the 18 SELIs. Figure 9 displays the most stakeholder associations between 
unique SELIs. 
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fiber of Stakeholders Per SELI 



Figure 9. Most Valued SELI Based on SSC-A CAO CONOPS and IPT 
Handbook 


The stakeholder to SELI analysis revealed that five of the 18 SELI listed in 
INCOSE’s SELI Guide version 2.0 were most important to the SSC-A stakeholders based 
on the SSC-A CAO CONOPS and IPT Handbook. The five SELIs were Risk Exposure 
Trends (Cost and Schedule), SE Staffing and Skills Trends, Process Compliance Trends, 
Technical Measurement Trends, and System Affordability Trends. These SELIs were 
evaluated closely to support SSC-A’s need for an automated SE / SE management risk 
visualization solution. SSC-A’s CAO CONOPS and IPT Handbook Stakeholder 
characterization/mapping with respect to SELIs is found in Appendix C. 

Further details on each of the SELIs selected for investigation in this report are in 
Chapter IV. 

2, Current External Tool Sources 

Current SE tool sources at SSC-A include software applications such as Navy- 
owned Navy Enterprise Resource Planning (NERP), Program Project Management and 
Control (P2MC), Civilian Personnel Information Management System (CPIMS), Mission 
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Alignment, Demand Signal, Team Assignment Agreement (TAA). COTS software such 
as IBM Rational Suite of tools, Atlassian Suite of tools, and several other less commonly 
used and possibly more specialized tools for specific modeling and simulation, 
maintenance and sustainment, or system configuration management are used as well. 

Each project customer or stakeholder places a varied set of requirements on IPTs. 
Some project stakeholders mandate the use of a particular SE tool to easily transfer or 
understand data based on their particular needs. Other project stakeholders allow the 
engineers or other IPT members to select and use any tool that they prefer but may or 
may not be willing to fund such use. This variation has caused a culture within SSC-A 
where there is little consistency of standard SE Data collection. In order for SSC-A 
stakeholders to have a complete visualization of SELIs, SE data would need to be 
normalized, e.g., presented in a consistent format, regardless of which of the various tools 
were being used. 

SSC-A is in the process of helping normalize the tools prior to tackling the effort 
of integrating and normalizing multiple SE Tool data streams. There is a current selection 
of tools identified to specifically provide the necessary SE Data fields required for 
analysis and visualization of SELI trends deemed most important to SSC-A leadership. 
Mapping the current planned tool selection roll out to the five selected SELIs, analyzed 
by this SEHV team, provided the basis for the data collection function of the SEHV 
analysis provided in the remainder of this report. 

3. Stakeholder Roles and Descriptions 

The following is a compressed view of the SSC-A CONOPS roles and 
descriptions. The detailed list with SELI mapping can be found in Appendix C. 

Command Senior Leadership Staff 

COMMANDING OFFICER (CO), EXECUTIVE DIRECTOR (ED) 

DIRECTOR OFMANAGMENTOPERATIONS 

DIRECTOR OF INTEGRATION AND EFFICIENCY 

INSPECTOR GENERAL (IG) 

Business Portfolio Board 

BUSINESS PORTFOLIO MANAGER (BPM) 
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DEPUTY PORTFOLIO MANAGER (DPM) 

PORTFOLIO SYSTEMS ENGINEER (PSE) 
SUBPORTFOLIO LEAD (SPL) 

INTEGRATED PRODUCT TEAM (IPT) LEAD 
IPT TECHNICAL LEAD 

5.0 Competency 

TIER 1 LOCAL COMPETENCY LEAD (LCL) 

LEAD SYSTEMS ENGINEER (LSE) 

CONTINUOUS PROCESS IMPROVEMENT (CPI) LEAD 
PROCESS OWNER 
PROCESS MANAGER 

COMPETENCY ADMINISTRATIVE SUPPORT 

TIER 2 COMPETENCY LEAD (CL) 

TIER 3 COMPETENCY MANAGER (CM) 

TIER 4 COMPETENCY SUPERVISOR (CS) 

FIRST-LINE SUPERVISOR 

CONTRACTING OFFICER'S REPRESENTATIVE (COR) 

Integrated Product Team 

PROJECT MANAGER 
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III. SEHV ACQUISITION OPTIONS ANALYSIS AND 
SELECTION 


Based on the above requirements and existing technology capabilities, the SEHV 
team discovered four potential acquisition options, one manual (the existing approach) 
and three automated, for providing the SSC-A SE Health and Visualization requirements. 
Each acquisition approach was evaluated using an AHP down select process which 
compared system functional area and cost as shown in the following figures below. The 
pairwise comparison charts and method to reaching the conclusion are also shown below 
in the following sections. Evaluation of each function included: Risks, Assumptions, 
Constraints, and Safety and Security concerns as explained in detail. 

A. ARCHITECTURE OPTIONS 

The two architectures options were a Manual Method and an Automated Method. 
The manual process of collecting data to provide SSC-A leadership information on data 
calls and project health is the existing process at SSC-A. The SEHV team explored and 
recommends an automated process to replace the manual process to improve efficiencies 
at SSC-A. Automated SEHV capability can be provided through several different 
acquisition options: Independent Commercial Off The Shelf (COTS) Tools (All COTS), 
Automated Centralized (GOTS) System and an Automated Hybrid Solution Set. 

i. Manual Method 

The manual method is defined as a non-automated collection of data. 
Management requires data on an individual electronic basis, requests this information 
from their subordinates; the subordinates request this information from their numerous 
team leads in a specific format. The team leaders collect and verify this information from 
each of their team members and send it up the chain of command for management 
analysis. The manual process of collecting SE data to search for indicators of project 
failures is currently the primary method used at SSC-A at this time. This existing manual 
method was included and analyzed to provide a “do nothing” option for a system solution 
and was used to compare against the other system options. 
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The benefit, of using the manual process, is that nothing at SSC-A with respect to 
data calls will have to change. Data calls will continue as is, and no training will be 
needed for this effort. The flexibility of requesting data, that suits the requestors’ needs, 
can easily be tailored during each data call cycle. Personnel have the freedom to use any 
tool with which they feel comfortable, excluding SSC-A mandatory tools such as NERP, 
Risk Exchange, or the Demand Signal Tool. 

There are also disadvantages to using this method. Manual data calls take 
significant personnel time. Typically, data calls contain instructions on what type and 
when data is needed. This methodology was generally inconsistent, so the resulting data 
obtained from these data calls would be outdated and stale. The delay is caused by 
personnel being involved with higher priority items, taking annual / sick leave, and on 
travel. 

Data collection is very troublesome when requesting information from personnel. 
This process requires personnel to capture and analyze required metrics manually. The 
cost for total man hours are increased due to the number of personnel involved in the 
process. Data storage of this information requires configuration management efforts from 
personnel which add to the costs. Displaying data trends manually requires time for 
personnel to manipulate the data into specific trends. 

2. Automated Method 

The automated method provides the capability instantaneously to collect data 
electronically using various SE Health Tools and software solutions. This method 
automates the collection of data to provide SSC-A’s management up to date information 
on project system engineering health as described in the simulation section of this report. 

B. ACQUISITION OPTIONS 

The automated architecture could be implemented through three acquisition 
options described below: Automated Independent COTS Tools (All COTS), Automated 
Centralized (GOTS) System, and an Automated Hybrid Solution Set. The manual option 
is the current system in place, and will not require any acquisition. 
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1. Automated Independent COTS Tools (All COTS) 

The Automated Independent COTS solution is defined as an automated process of 
collecting, storing, analyzing, and displaying the data using one complete or a set of 
existing commercial off the shelf solutions. The Automated Independent COTS option 
was identified to shorten development time and to leverage industry standards. This 
solution would inherently insert potentially challenging upgrades and licensing issues, as 
discussed clearly and in depth by Dr. David A. Cook in “Issues to Consider before 
Acquiring COTS. ” However, having the mature technology available and being able to 
rapidly provide SE Health and Visualization indicators rapidly may be worth managing 
the risk. 

The COTS tools and software upgrades would be maintained by the manufacturer. 
This would reduce the responsibility on the government for maintenance efforts with the 
exception of ensuring basic preventive maintenance practices. 

The disadvantage of having an Automated Independent COTS solution would be 
that personnel are required to login to each one of the COTS tools to obtain the 
information desired by leadership. Manual processes may still have to take place to 
achieve the visualization of trends that leadership would want to analyze. There may be 
multiple software suites with little to no capabilities to display desired SELI visualization 
trends. Initially, the COTS tools would have to be procured and training of the tools 
would have to be coordinated; both would be front-loaded cost drivers. If the COTS tools 
are not all integrated by one manufacturer, there could be significant challenges to 
coordinate testing and approval prior to enabling the software to be automatically 
updated. 

Data collection for Automated Independent COTS Tools would be possible for 
personnel to obtain information from a specific set of tools. This would also involve 
personnel having to gather and input the data; however, input of the data would be in 
various specified independent software suites. Analyzing this data would be easier since 
certain COTS tools are already mandated throughout the command. Therefore, having 
each of the related data fields automatically linked would reduce integration and analysis 
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delay challenges. The cost for this solution would include software, hosting environment, 
licenses, training, and man hours spent manually gathering data from tool suites. Data 
commitment would still require manual configuration management. Independent tool 
suites have the capability to display trends, but the integrated COTS combined display 
provides the capability to meet stakeholder visualization requirements. 

2, Automated Centralized GOTS System 

An automated centralized Government Off The Shelf system is defined as an 
automated process of collecting data using all software that is fully designed and 
developed “In-House.” This type would require a plethora of programming and software 
design labor hours, as well as a suitable system maturity process that would allow for 
acceptable software modifications to occur as lessons are learned and requirements or 
system needs change. This option would require longer wait time for realization to 
provide results than any other option. 

A benefit of having a GOTS System would include the ability to tailor the 
software to the exact needs of producing specific SELI trends. A web graphic user 
interface could be produced requesting planned data from the SE or SE Manager. That 
planned data could be compared with the actual data of the GOTS system that would pull 
from the SSC-A mandatory databases already in place. The planned versus actual data 
measures would show trends to determine overall project risk. Other benefits are the 
ability to rapidly direct work and change priorities without going through the contract 
process thus increasing the time for development. Quality Assurance, clean 
documentation, and control over the development of the product would prove 
advantageous for effectively saving cost. This proprietary in-house software would 
belong to SSC-A. 

The disadvantages of the GOTS System include the time required for developing 
this in-house software and maintaining the software throughout the system’s life cycle. 
Upgrades and updates of in-house software requires in-house personnel effort to perform 
the maintenance requirements. Subject Matter Experts (SME), for this tool, would have 
to be trained to provide the level of support required by users that may have trouble using 
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this type of system. Understanding that the advantages and disadvantages of GOTS could 
be costly, further study should be conducted using a cost-benefit analysis of this solution. 

The data collection for the GOTS System would have to be tailored to support the 
required measures needed by leadership. Although, analyzing and capturing required 
metrics is possible with in-house developed software, developing a tool to capture, store, 
analyze and display will require a extraordinary degree of effort. Data commitment 
would be developed and tailored to the command’s needs and displaying trends would 
need to be incorporated during development to show SELI trends. 

3. Automated Hybrid Solution Set 

A Hybrid System is defined as an automated process of collecting data using both 
COTS and GOTS software solution. The Hybrid System solution concept is the attempt 
to take the benefits of existing COTS Systems Engineering Tools and integrate them with 
In-House Tools. This option would help reduce time to provide a minimum visualization 
capability, but could then be modified to display specific visualizations. Inherently, this 
option would bring additional risks due to interface challenges that could occur as COTS 
are upgraded, compared to the other options. 

The benefit of having a Hybrid System would include having middleware to 
integrate different tool suites, both COTS and GOTS. The Hybrid solution would have 
interoperability between the tools by harvesting the required data elements to produce 
visualization. The methodology of capture, store, analyze, and display would be used to 
effectively produce SELI trends. The concept of having SE / SE managers having the 
ability to use tools without being interrupted by data calls would be met. This solution 
would be the “middle man” to get the data elements from the tools that the SE / SE 
managers desire. This solution would also store these data elements without having 
personnel manually performing configuration management on historical artifacts. It 
would have the analytical features to perform basic calculations automatically for the 
visualization of this data. This would reduce the need for personnel manually performing 
these tasks as well as keep the integrity of the information. Up-to-date information would 
be the result of the Hybrid System. The disadvantage of the Hybrid solution includes an 
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upfront initial investment. Material and service procurement efforts have to take place to 
obtain the COTS tools as well as creating the Hybrid System. 

The Hybrid solution would allow data collection to take place on command 
required tool suites. This provides little disruption to the SE / SE managers while data is 
gathered. Mandated tool suites for SSC-A and the use of developed software capturing, 
storing, analyzing, and displaying data would provide analytical data for the command 
performing SE activities. The initial cost would involve procurement of COTS software 
and the development of the “middleware” software. 

C. ANALYTIC HIERARCHY PROCESS (AHP) EVALUATION CRITERIA 

The architecture functions were evaluated within the AHP by analyzing the 
Assumptions, Constraints, and Safety and Security certification concerns identified in the 
below paragraphs for each system architecture. 

1. Assumptions 

The below assumptions were used when rating the architecture options during the 
AHP evaluation. 

Manual Method 

It was assumed that with the existing manual SE data input and display process, 
SE indicators were not available fast enough to prevent project failures or issues. It was 
also assumed and believed that this method took the longest time from data input to 
visualization. This was further evaluated in a modeling and simulation comparison with 
the automated option. 

Automated Independent COTS Tools 

It was assumed that for the All COTS solution not all desired SE data input or 
resulting SE health visualizations would be provided or even available. Further research 
would be required for each of the identified COTS tools to perform a thorough 
comparison of performance capabilities and limitations. 
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Automated Centralized GOTS System 


It was assumed that the Automated GOTS solution would provide all initially 
identified capabilities. This solution would allow for SE data input into the interface 
method desired and required by the stakeholders, which would be a balance that often 
challenges the use of a system. It was also assumed that this solution would be modifiable 
by in-house support, and that training and help desk capabilities would be available as 
required. 

It was further assumed that there could be an inherent loss of creativity, over time, 
with respect to generating a visual or even collecting SE data when choosing this system 
alternative. Without the feedback from industry on new developments, the current 
development and maintenance workforce would not see a need or even the possibility for 
improvement. 

Automated Hybrid Solution Set 

With the Hybrid System solution, it was assumed that the initial capability 
requirements would be met and the solution realized quickly. Although it likely required 
more time than with the All COTS solution, it was assumed multiple COTS product tools 
would be integrated with in-house developed tools to provide the data input, analysis, and 
visualization of SE leading health indicators. 

2. Constraints 

Each SEHV system architecture alternative has a different set of constraints that 
may impact their suitability to meet stakeholder requirements. For each of the options, 
specific constraints are identified as related to the system functions or other suitability 
criteria. 


Manual Method 

The existing manual process is constrained by current manpower availability. As 
SSC-A requests a change to an SE Process, the SE data input personnel require additional 

operational downtime to learn the new process and work out the new format of 
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information required for delivery. This often comes as a secondary requirement to the 
project effort itself, and therefore is not often implemented well. This labor constraint can 
cause schedule delays, reduced quality or cause incomplete products. 

Automated Independent COTS Tools 

The major constraint with the All COTS system solution was that upgrades occur 
when the vendor require thus making it impossible to maintain control over the 
environment. This allowed for interoperability challenges and training requirements 
depending on the magnitude of each applied change after the upgrade. 

Automated Centralized GOTS System 

Contrary to the All COTS constraints, this option would deploy upgrades, but 
would be able to do so only with SSC-A approval. With this control, there were still 
Safety and Security constraints that would force immediate upgrades to protect resources. 

Automated Hybrid Solution Set 

Similar to both the All COTS and in-house options, constraints on the Hybrid 
solution seem not to only have the sum of each individual option, but also included the 
integration constraints. As COTS upgrades occur, SSC-A would be forced to modify its 
in-house systems to continue to provide full functionality of the SEHV. 

3. Safety and Security Certifications 

Safety and Security Certification and/or Accreditation considerations play a large 
part of an IT system. Information Assurance (IA) requirements are identified in the DOD, 
based on the level of sensitivity of the data being stored or aggregated, to ensure data 
availability, authenticity, and other security best practices. Among the SEHY architecture 
options listed, the Manual Method is the only one that does not require direct IT 
Certification. However, depending on the implementation of each option, different safety 
and security concerns may apply to each. 
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Manual Method 


Safety and Security measures for the existing manual process were already in 
place and being performed. Replacement of this method with one of the other three 
methods does not guarantee that those measures would still apply. Typically, each new 
option posed a new set of Safety and Security concerns to be evaluated. 

Automated Independent COTS Tools 

Safety concerns are addressed in the development of the COTS tools. Operational 
safety concerns would still need to be reviewed and evaluated. 

Security certifications would be a challenge depending on the type of data the 
COTS allowed for storing or processing. If the data was sensitive to business operations, 
then special security concerns would need to be identified. Specifically, internal security 
measures should either allow or prevent SE project collaboration where more than one 
person could view or edit project data. However, there was a need to virtually host the 
systems behind network security layers or other types of IT security features approved 
prior to implementation and use. The most common government security requirement for 
a software system of these types would follow DOD information assurance requirements 
for Certification and Accreditation for use on a United States military network. Data 
commitment applications, SE tools, and visualization software would have to be 
approved and maintained to ensure the current software patches are being used. 
Vulnerabilities to software used in the SEHV solution would need to be updated and 
scanned on a consistent basis to satisfy IA security requirements. 

Automated Centralized GOTS System 

Safety and security considerations for this option would need to be reviewed for 
consistency with policy. They should be easier to comply with due to internal 
development, having access to the code and to the support personnel performing system 
modifications. This solution would cause a different level of safety and security concerns 
but they are expected to be resolvable faster than with the All COTS solution. This is due 
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to design, development, programming being performed internally, and allowing direct 
access to experts able to perform required modifications. 

Automated Hybrid Solution Set 

Security concerns for the Hybrid system option include coordination for the 
integration of COTS and an internal government self-development effort. COTS pieces 
would require their specific IA Accreditation as well as a separate certification of the 
GOTS components. Database software, SE / SE management tools, visualization 
software, as well as a graphic user interface (GUI) would have to be kept up to date with 
the latest IA security patches. Hosting environments would have to be scanned with IA 
tools to determine potential vulnerabilities and COTS and GOTS software would have to 
be evaluated by the SSC-A IA team to ensure there are no software security risks. 

D. ANALYTIC HIERARCHY PROCESS 

An AHP hierarchy is a structured means of modeling the decision at hand. It 
consists of an overall goal, a group of options or alternatives for reaching the goal, and a 
group of factors or criteria that relate the alternatives to the goal. It is a structured 
technique for organizing and analyzing complex decisions, based on mathematics, 
psychology and engineering judgment. It has particular application in group decision 
making (Saaty 2008a). 

Rather than prescribing a “correct” decision, the AHP helps decision makers find 
one that best suits their goal and understanding of the problem. It provides a 
comprehensive and rational framework for structuring a decision problem, for 
representing and quantifying its elements, for relating those elements to overall goals, and 
for evaluating alternative solutions {Saaty 2008b). The SEHV team used the AHP to 
evaluate four acquisition alternative ways of reaching the goal, and five criteria against 
which the alternatives need to be measured (after Saaty 2008b). 
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Overview 


An evaluated selection process was conducted to determine the best analysis 
option to support the SEHV capability. The high level requirements stated that SSC-A 
required a system to provide SSC-A leadership with a tool to monitor SE program health 
in support of the warfighter. An AHP was performed to assist in making this 
determination. The following decision scenarios were considered in selecting the best 
acquisition option: 

• SSC-A was in need of a solution for visibility into the SE health of its 
programs 

• Acquisition procurement option had to be chosen for the SEHY capability 

• Only two architecture options were available; Manual and Automated 

• Choice would drive direction of SEHV system 

• The team considered the following functions: Data Collection, Capture 
and Analyze Required Data Elements, Cost Effectiveness, Data Storage, 
and Data Trends Display 

The SEHV team collected the acquisition options and the consideration criteria 
and conducted a pairwise comparison of alternatives with respect to each other. The 
priority vector of each criterion was calculated and then multiplied against each 
alternative to reach a solution. Figure 10 shows the SEHY solution strategy where 
supporting data and figures used to reach this decision are presented following this 
image. 
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Figure 10. SEHV Analytic Hierarchical Process Diagram (after Green 2013) 


2. Alternatives Comparison 

Initially, a decomposition of decision criteria into comprehended sub-factors (i.e., 
data collection, capture and analyze required data elements, cost, data storage, and 
display data trends; each of which was analyzed independently) was performed (after 
Saaty 2008b). Utilizing engineering judgment, the team set levels of importance values 
from which to compare the criteria for fair competition. A pairwise comparison of the 
acquisition alternatives was performed to reach a formal conclusion of the best alternative 
for the SEHV solution strategy. Figure 11 displays the pairwise comparison matrix used 
for this project. The relative importance is shown with an associated importance value 
that aided in determining hierarchy of elements according to their importance 
comparatively. 
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Relative Importance 

Value 

Equal Importance/Quality 

1 

Somewhat More Importarrt/Better 

3 

Definitely More Important/Better 

5 

M uch M ore Important/Better 

7 

Very Much More Important/Better 

9 


Figure 11. Pairwise Comparison Matrix (from Green 2013) 

Once the hierarchy was built, the team systematically evaluated the various 
elements by comparing them to one another two at a time, with respect to their impact on 
an element above them in the hierarchy. In making the comparisons, the team typically 
used their engineering judgments about the elements’ relative meaning and importance. 
Using this pairwise comparison matrix as the foundation, a criteria comparison was 
performed to determine the priority vector of each element compared to one another. The 
hierarchy was constructed and analyzed through a series of pairwise comparisons that 
derived numerical scales of measurement. The criteria were pairwise compared against 
the goal for importance. The alternatives were pairwise compared against each of the 
criteria for preference. The comparisons were processed mathematically, and priorities 
(priority vectors) were derived for each (after Saaty 2008b). To show the element as less 
important to the competitor, the inverse was taken. Figure 12 displays the resulting 
outcome. 
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Figure 12. Criteria Comparison (after Green 2013) 


To achieve the relative weight of a criterion or alternative, like probabilities, 
priorities are absolute numbers between zero and one, without units or dimensions. A 
criterion or alternative with priority .200 has twice the weight in reaching the goal as one 
with priority .100, ten times the weight of one with priority .020, and so forth. Depending 
on the problem at hand, “weight” can refer to importance, or preference, or likelihood, or 
whatever factor is being considered (after Saaty 2008b). The SEHV team refers to 
weight as preference for this capstone. 
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Priorities were distributed over the hierarchy according to its architecture, and 
their values depended on the information entered. Priorities of the Goal, the Criteria, and 
the Alternatives are intimately related, but needed to be considered separately. By 
definition, the priority of the Goal is 1.000. The priorities of the alternatives always add 
up to 1.000 (after Saaty 2008b). The SEHV team performed a comparison of acquisition 
options with respect to each alternative. Each comparison was concluded with the 
determination of a respective priority vector. Figures 13-17 provide supporting detail for 
the overall decision. 
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Figure 13. Alternatives with Respect to Data Collection (after Green 2013) 
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Figure 14. Alternatives with Respect to Capture and Analyze Required Data 
Elements (after Green 2013) 
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Figure 15. Alternatives with Respect to Cost Effectiveness (after Green 2013) 
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Figure 17. Alternatives with Respect to Data Trends Display (after Green 2013) 


3. Conclusion to Solution Strategy 

For clarity, in this SEHV strategy, the goal is to select the best acquisition option; 
the criteria were the data collection, capture and analyze required data elements, cost, 
data storage, and display data trends capabilities; and the alternatives were the Manual 
Method, Automated COTS, Automated GOTS and Automated Hybrid options. To reach 
a determination of the best acquisition option for recommendation, a criterion versus goal 
comparison had to be completed. The total column value of each criteria element was 
divided into each element respectively. Figure 18 depicts where those values were then 
added (across row) to obtain the priority vector to be used to calculate the priority of each 
criterion alternative. 


55 




Data Collection 

Capture & 
Analyze 

Cost 

Data Storage 

Data Trends 

Display 

Data Collection 

1 

.20 

.14 

3 

5 

Capture & 
Analyze 

5 

1 

.14 

9 

5 

Cost 

7 

7 

1 

9 

7 

Data Storage 

.33 

.11 

.11 

1 

.33 

Data Trends 

Display 

.20 

.20 

.14 

3 

1 

Totals 

13.53 

8.51 

1.53 

25 

18.33 



Data 

Collection 

Capture & 
Analyze 

Cost 

Data Storage 

DataTrends 

Display 

P.V. 

Data 

Collection 

.07 

.02 

.09 

.12 

.27 

.11 

Capture & 
Analyze 

.37 

.12 

.09 

.36 

.27 

.24 

Cost 

.52 

.82 

.65 

.36 

.38 

.55 

Datastorage 

.02 

.01 

.07 

.04 

.02 

.03 

DataTrends 

Displ^ 

.01 

.02 

.09 

.12 

.05 

.06 


Figure 18. Criteria versus Goal Comparison (after Green 2013) 


Figure 18 displays the calculations that provide necessary information to 
determine the priority of each alternative with respect to criteria versus goal. Figure 19 
shows the calculation matrix of each criterion hosting each alternative. The resulting 
answers were added together in a priority vector matrix, Figure 20, of criteria with 
respect to each acquisition option to conclude the final goal of the SEHV solution 
strategy. 
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Criterion 
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A 
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Figure 19. Priority Relationship Calculation Matrix (after Green 2013) 
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Figure 20. Criterion Priority with Respect to Acquisition Option (after Green 
2013) 
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Based on the team’s comparison, the Automated Hybrid Solution Set was the 
preferred alternative with a priority of .459. The Hybrid Solution Set was preferred over 
the other alternatives of the Automated Centralized System with a .378, then the 
Automated Independent COTS Tools with a .115, and finally the Manual Method with a 
.056, as seen in Figure 20. 
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IV. SELECTED SELI DETAILS 


Five selected SELIs are detailed below and describe how SSC-A could tailor them 
for use with their current processes and systems. Each SELI discusses the associated 
processes, data collection procedures, data storage information, data analysis procedures, 
historical data information, and sample visualizations. Potential issues regarding 
implementation of these SELIs are also identified. 

A. SELI 9, RISK EXPOSURE TRENDS 

Described in INCOSE’s SELI Guide version 2.0, the Risk Exposure Trends 
indicator depicts whether the project is effectively managing the project risks shown by 
predicted exposure ratings over time. Assessments can be made from trends that expose 
impacts to the project. This trend can illustrate whether a project is adequately managing 
exposed risks. The measurable concept includes having the capability to assess a 
project’s effectiveness in managing and mitigating risks. This trend also promotes 
identifying risks that can severely impact the project’s success. Managing the exposed 
risk can also be viewed when mitigation tasks are performed and referred to within the 
trend at a later date to see if the mitigation effort worked. This can indicate whether the 
mitigation of the risk was effective. If a particular mitigation is executed and resulted in 
extreme costs, this may indicate that another alternative may need to be considered if the 
risk were to appear again. 

To properly use this trend effectively, a continuous risk management practice 
should be in place along with a repository that can store historical data. An active risk 
management habit will show potential risk for a project. 

Risks related to the SEHV systems or applications were identified and 
documented based on the methodology in the INCOSE SELI 9 as well as NIST SP 800- 
30, Risk Management Guide for IT Systems. IT systems or application weaknesses were 
identified on an associated plan of action and milestones (POA&M) and tracked in 
accordance with SEHV POA&M guidelines. Appropriate protective measures were taken 
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to safeguard sensitive IT and SE system for application weaknesses or vulnerabilities 
from unauthorized disclosure. 

1. Associated Process 

Risk Exchange is currently being used at SSC-A, Charleston. SSC-A, New 
Orleans (NOLA), is currently using Excel spreadsheets. Table 3 depicts other tools that 
are in use to manage risk. 


Table 3. Additional Risk Management Tools 


Product 

Company 

Focus 

CRAMM 

Insight Consulting Ltd. 

Government, Public Sector 

CORA 

International Security 
Technology INC 

Telecom, Logistics, 
Government, IT 

COBRA 

C&A Systems Security Ltd. 

Enterprise 

Risk Check 

Norman Security Solutions 

Enterprise 

Risk PAC 

CSCI Inc. 

Business Continuity 

Risk Watch 

Risk Watch Inc. 

HIPAA, DITSCAP, 

NIACAP 

The Buddy System 

Alion Science & 

Technology 

IT 


2. Data Collection Procedure 

Extracting high, medium, and low risks from the Risk Exchange tool is required 
to produce risk exposure trends. Obtaining risk data on a monthly basis will ensure up to 
date trends are produced. A historical database is needed to show these trends over time. 
Table 4 displays the data collection procedure used for risk exposure. 
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Table 4. Risk Exposure Data Collection Procedure 


Base Measure 

Data Collection Procedure 

Frequency of Data 
Collection 

Monthly 

Responsible 

Individual 

Project Manager 

Activity in which 
collected 

Extracting high, medium, and low risks from the Risk Exchange 
tool 

Potential Sources 
of Data 

Risk Exchange 

Typical Tools used 
in Data Collection 

Online GUI 

Verification and 
Validation 

Reviewed Monthly 

Repository for 
Collected Data 

Suggested: Historical Database 


3. Data Storage 

Current risk data is stored in Risk Exchange and other risk tools. Depending on 
the SSC-A accepted risk data collection frequency, risk data should be pulled from Risk 
Exchange, time stamped, and stored in the SEHY historical database. 

Table 5, which provides data elements, is suggested for collecting and storing 
Risk Exposure data. 
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Table 5. Risk Data Elements 


Inputs/Outputs (Data Elements) 
Required 

Tool or Data 

Sources 

Metric/Formula used With 

List of Portfolio Names for Risk 
Exchange 

Risk 

Exchange 

% of Portfolio’s using Risk Tool 

List of Project Names for Risk 
Exchange 

Risk 

Exchange 

% of Projects using Risk Tool 

Count of CURRENT Risks LOW 
Criticality to Cost/Schedule/ 
Performance, per Project, per 

Portfolio 

Risk 

Exchange 

Low Risk Trends 

Count of CURRENT Risks 

MEDIUM Criticality to Cost/ 
Schedule/Performance, per Project, 
per Portfolio 

Risk 

Exchange 

Medium Risk Trends 

Count of CURRENT Risks HIGH 
Criticality to Cost/Schedule/ 
Performance, per Project, per 

Portfolio 

Risk 

Exchange 

High Risk Trends 

Count of PREVIOUS Risks LOW 
Criticality to Cost/Schedule/ 
Performance, per Project, per 

Portfolio 

SEHV 

Analysis and 
Visualization 

Server 

Low Risk Trends 

Count of PREVIOUS Risks 

MEDIUM Criticality to Cost/ 
Schedule/Performance, per Project, 
per Portfolio 

SEHV 

Analysis and 
Visualization 

Server 

Medium Risk Trends 

Count of PREVIOUS Risks HIGH 
Criticality to Cost/Schedule/ 
Performance, per Project, per 

Portfolio 

SEHV 

Analysis and 
Visualization 

Server 

High Risk Trends 
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4. Data Analysis Procedure 

SEHY stored risk data will consist of the Low, Medium, and High risks per 
project. Analysis of these risks will initially involve only the high level counts of each, to 
be graphed with the current counts, which should enable the visualization of increasing or 
decreasing trends in Risk Exposure. Table 6 displays the data analysis procedure with 
respect to risk exposure. 


Table 6. Risk Exposure Data Analysis Procedure 


Indicator 

Process Compliance Trends 

Frequency of Data 
Reporting 

As requested by authorized personnel 

Responsible 

Individual(s) 

Project Manager, IPT Lead 

Code 01B Director of Management Operations 

Code Oil Inspector General 

Business Portfolio 

Business Portfolio Management (BPM) 

Deputy Portfolio Manager (DPM) 

SubPortfolio Lead (SPL) 

Integrated Product Team (IPT) Lead 

IPT Technical Lead 

Integrated Product Team 

Project Manager 

Activity which 
Analyzed 

Observing Risk trends and mitigation efforts 

Source of Data for 
Analysis 

Risk Exchange 

Tools Elsed in 
Analysis 

Risk Exchange 

Review, Report, or 
Elser 

Review 
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5. 


Historical Data 


Many programs have historical risk data that may or may not be easily 
comparable to the SEHV suggested Risk Exposure trend data. The SEHV database would 
need to store newly collected Risk data elements from Risk Exchange or other risk tools, 
on either weekly basis or other SSC-A approved frequency. This data would then be time 
stamped and used as historical data for display and comparison against current data as 
requested by SSC-A stakeholders to view SE Risk leading indication of project health. 

6, Sample Visualizations 

Figure 21 displays Risk Exposure Trends in low, medium and high priority risks 
over time with respect to cost and schedule. A steady decline of risks would indicate a 
project was performing risk management practices. Leadership and stakeholders of a 
project would discuss these trends during regularly scheduled reviews. 


Risk Exposure Trends 



Figure 21. Risk Exposure Trends (from INCOSE et al. 2010) 
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B. SELI11, SE STAFFING AND SKILLS TRENDS 

SE Staffing and Skills Trends are used to see how many SE activities are being 
performed on a project. The amount of SE activities will dictate the number of personnel 
or needed personnel for a project. To adequately keep a project going, especially for a 
large Acquisition Category I (ACAT1) program, continuous SE activities are needed and 
required. SSC-A has identified continuous management functions such as risk 
management, requirements management, interface management, configuration 
management, technical data management, technical assessment, decision analysis, service 
management, product support management, quality management, lessons learned 
management, business resource & financial management, contract management, science 
and technology management, and human resource management that must be conducted 
throughout the life cycle of the project. 

The SE Staffing and Skills Trends SELI would be used to identify if the project 
has the right manpower resources to accomplish project objectives. The need for this type 
of trend at SSC-A is critical to determine if the amount of SE a project needs aligns with 
the personnel resources on that project. Personnel volatility can also indicate a potential 
risk of cost overruns, and schedule slippages. New SE personnel always undergo a 
learning curve where understanding the project, and knowing who are the key players 
takes time. Personnel retention is favorable if the project is performing well and the job is 
being completed. High personnel turnover can also indicate that there may be a much 
larger invisible problem on the project. A project with unrealistic goals, continuous 
punishable repercussions on team members by management, hostile work environment 
and other negative project traits can lead to team members leaving the project. 

These trends can also display if a project is even performing certain SE efforts. 
The lack of performing critical SE functions can give a strong indication of project 
failure, especially when a project is not doing well to begin with. When a project is not 
doing well, an aspect to consider is if the team members of the IPT have the appropriate 
knowledge, skills, and abilities (KSAs) to perform SE tasks. Training can help mitigate 
some of these KSA deficiencies; however each person has a learning ability that may be 
greater than or less than the next person. 
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The leading insight provided for this SELI would illustrate a right balance of SE 
staffing for successful projects versus unbalanced SE staffing for a struggling project. 
Although SSC-A currently does not track this type of measure, the sooner this SELI is 
being utilized and tracked, the sooner historical data can be created to display such 
trends. SSC-A does have a Competency Development Model (CDM) for personnel; 
however soon after the practice was performed, the CDM data gathering stopped. This 
effort will need to be rekindled to depict the skill levels of personnel especially with SE 
efforts. SE Staffing and Skill Trends display gaps and / or shortfalls of SE effort with 
respect to IPT members KSAs. Poor project outcomes can result from inadequate SE 
staffing and skillsets. 

Planned SE efforts vs actual SE efforts can be measured to display trends over 
time. If the planned SE efforts outweigh the actual SE efforts for a troubled project, 
perhaps either there is not enough manpower within the organization or SE / SE 
management practices are not being considered as important. Since different amounts of 
SE efforts are considered during each phase of the project life cycle, planned SE effort 
projections are needed to provide early indication that insufficient personnel risks 
inadequate performance of SE practices. Unit of measures for this particular SE can be 
portrayed as SE man-hours associated with the type of SE process. Planned values of SE 
man-hours can enforce the need of either fulltime staff to perform SE daily or part-time 
staff that perform these functions a number of hours throughout the week as a collateral 
duty. Figure 22 displays a visual depiction of this concept. 
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Figure 22. Planned SE Effort versus Actual SE Effort 


SE activities can fluctuate throughout the project life cycle. An example would 
include volatile or unclear requirements during the planning phases which numerous 
man-hours for SE requirements management is needed vice when the project is within the 
production and deployment phase of the project life cycle. Another example would 
include risk management being a critical need for a struggling project during a period of 
time when a plethora of high risks are realized vice another period of time of only having 
minimally low and medium risks. 

Measurement methods for this SELI include SE effort hours for individual SE 
process areas with respect to time. Counting the planned effort hours and comparing with 
the actual effort hours spent on SE will show possible deficiencies in any particular SE 

activity on a project. An array of data involving the planned vs. actual SE man-hours is 
67 







used to display the trends in the form of bar graphs. SE / SE management will input the 
planned values for each SE process area at least monthly to keep an updated SE effort 
profile on a project. Failure to do so should involve leadership intervention to mitigate SE 
management shortfalls within a project. 

NERP has Global Work Breakdown Structure (GWBS) elements assigned to 
Network Activities (NWAs) to which government personnel charge their time. These 
GWBS elements specify SE activities such as Studies & Design, Architecture, Human 
Systems Integration (HSI), Technical Authority, Certification Authority, Systems, 
Engineering Management, Requirements Analysis, Configuration Management, and 
Logistics Engineering. Although not all SE efforts are listed under the GWBS, additional 
SE activities could be added to the GWBS or rolled up into other labeled criteria. 

SE / SE management staffing needs, actual available personnel, and total 
workforce trends can reveal how much emphasis SSC-A has on SE / SE management 
staffing and skills. Information such as the length of time a SE / SE management is 
needed for a project can determine availability of personnel at a given point in time. This 
measure could also illustrate when projects come to an end, an actual surplus of 
personnel within the SE field. Having a well-balanced workforce is essential for the 
success of a project, program, command, or any Government / Military organization. 
Tools such as the Demand Signal Tool and Total Workforce Management System 
(TWMS) should be used to obtain the appropriate data to display these trends. 

SSC-A has a staffing tool (Demand Signal Tool) that is used to determine a 
project’s needs in regards to manpower. The Demand Signal Tool allows projects to 
submit their needs for staffing. This tool can provide data that shows how many resources 
the command needs with respect to a particular position. Common needs throughout 
projects can reveal a deficiency for a particular type of work within the command. If the 
need is high enough, hiring action should be considered to fill in the gaps where SE / SE 
management is critically needed. Exporting this data to an Excel spreadsheet is a feature 
of the Demand Signal Tool. Data can be filtered into different areas of interest once the 
data is exported. Extracting this data from the Demand Signal Tool can provide valuable 
information to the SEHV system. Information such as the amount of SE / SE 
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management personnel the command needs by IPT, portfolio, competency, and sub¬ 
portfolio. This data, for needed personnel, would be updated to reflect the status of the 
Demand Signal itself. This updated data would fuel the SEHV system for visualization 
display to the command. 

TWMS is used to include skillsets, education, training, experience, pay bands, 
military / veteran information, and position. Within SSC-A’s Competency Aligned 
Organization (CAO), personnel are aligned within competencies associated with the type 
of work that is being performed. This type of methodology would include knowing what 
competency to choose for a particular type of work. For instance, the 5.2 competency is 
primarily for SE / Industrial Engineering. The 6.3 competency is strictly for program 
management. The 4.3 competency is labeled as the logistics competency. Aligning 
similar skillsets to a specific group is a good way to organize personnel. Unfortunately, 
this methodology restricts personnel to performing only the advertised competencies and 
responsibilities. Although personnel may have the expertise and capabilities to perform 
multiple functions, that same personnel is limited on what they can or cannot do when 
placed in a specific competency. A person with multiple capabilities such as SE 
management and logistics assigned to the 4.3 competency would be limited to performing 
logistics functions rather than performing both SE management and logistics tasks. The 
organization should include primary, secondary, and even other skillsets that personnel 
may have in order to economically provide the right expertise for the right type of work 
demanded needed. This data would be extracted to depict trends relating to staffing and 
particular skill sets. These KSAs could be displayed as bar graphs for the amount of 
available personnel with specific skill sets or even reveal a deficiency in a particular skill 
group. These data elements should be compared with the Demand Signal Tool 
information on the number of needed SE / SE management resources. 

SE Staffing and Skill Trends display a project’s effort in regards to SE. Planned 
values should be reviewed monthly by the IPT Lead. If actual SE hours are not meeting 
the planned or expected amounts of SE required, this could allow leadership to intervene 
to promote the execution of needed SE activities. Another way of using SE Staffing and 
Skill Trends would be to show how many SE / SE management personnel are needed in 
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an organization compared to the actual SE / SE management personnel available. A high 
demand of personnel could trigger the Tier 3 Competency Manager to begin the hiring 
action process. The IPT lead could also use this SELI trend to pick up available personnel 
for their IPT. 


1. Associated Process 

The NERP, TWMS, and Demand Signal tools are used to capture data for this 
SELI. The data is captured real time and the information would be evaluated monthly, 
and as needed when the information is being requested. 

2, Data Collection Procedure 

Extracting data from NERP, TWMS, and the Demand Signal Tool is required to 
produce SELI trends. Obtaining SE staffing and skills data on a monthly basis will 
ensure up to date trends are produced. A historical database is needed to show these 
trends over time. Table 7 displays the data collection procedure for SE staffing and 
skills. 


Table 7. 

SE Staffing and Skills Data Collection Procedure 

Base Measure 

Data Collection Procedure 

Frequency of Data 
Collection 

Monthly 

Responsible 

Individual(s) 

Project Manager, IPT Lead 

Activity in which 
collected 

SE Planned vs SE Actual (Performed on a Project) 

Manpower Needs vs Available 

Potential Sources of 
Data 

NERP, TWMS, Demand Signal Tool 

Typical Tools used in 
Data Collection 

Excel 

Verification and 
Validation 

Reviewed Monthly 

Repository for 

Collected Data 

Suggested: Historical Database 
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3. 


Data Storage 


SE Staffing and Skills trend data should be stored in a historical database. This 
historical data can be pulled and compared to actual current data from each of the tools 
used within SSC-A. Most SSC-A projects use the same tools to store their project staffing 
requirements and actual usage or effort spent on the project over time. An interface will 
need to be able to collect the actual effort data from each tool, per skill, per project, and 
per portfolio at the frequency interval determined, and store within the historical 
database. Table 8 is suggested for collecting and storing SE Staffing and Skills data. 


Table 8. Staffing and Skills Data Elements 


Inputs/Outputs (Data Elements) 
Required 

Tool or Data 

Sources 

Metric/Formula used With 

List of Portfolio Names for SE 
Staffing & Skills Trends 

NERP 

% of Portfolio’s using SE Staffing 
& Skills Tool 

List of Project Names for SE Staffing 
& Skills Trends 

NERP 

% of Projects using SE Staffing & 
Skills Tool 

List of Skill Areas per Project, Per 
Portfolio 

NERP 

Compare to Standard SE Template 
Required for that IPT or Project 
Type 

CURRENT SE Effort PLANNED in 
hours, per skill, per project, per 
portfolio 

NERP 

SE Effort per Project Trend 

CURRENT SE Effort ACTUAL in 
hours, per skill, per project, per 
portfolio 

NERP 

SE Effort per Project Trend 

CURRENT SE Effort AVAILABLE 
in hours, per skill, per project, per 
portfolio 

TWMS 

SE Effort AVAILABLE minus 
PLANNED equals SSC-A 
Overhead Surplus Effort 

CURRENT SE Effort REQUESTED 
in hours, per skill, per project, per 
portfolio 

Demand 

Signal Tool 

SE Effort REQUESTED minus 
OBLIGATED equals Effort 
Unfilled but Available 

CURRENT SE Effort OBLIGATED 

TAA Tool 

SE Effort OBLIGATED minus 
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in hours, per skill, per project, per 
portfolio 


ACTUAL equals Effort Project 

Gap Unplanned 

PREVIOUS SE Effort PLANNED in 
hours, per skill, per project, per 
portfolio 

SEHV 

Analysis and 
Visualization 

Server 

SE Effort per Project Trend 

PREVIOUS SE Effort ACTUAL in 
hours, per skill, per project, per 
portfolio 

SEHV 

Analysis and 
Visualization 

Server 

SE Effort per Project Trend 

PREVIOUS SE Effort AVAILABLE 
in hours, per skill, per project, per 
portfolio 

SEHV 

Analysis and 
Visualization 

Server 

SE Effort AVAILABLE minus 
PLANNED equals SSC-A 
Overhead Surplus Effort 

PREVIOUS SE Effort REQUESTED 
in hours, per skill, per project, per 
portfolio 

SEHV 

Analysis and 
Visualization 

Server 

SE Effort REQUESTED minus 
OBLIGATED equals Effort 
Unfilled but Available 

PREVIOUS SE Effort OBLIGATED 
in hours, per skill, per project, per 
portfolio 

SEHV 

Analysis and 
Visualization 

Server 

SE Effort OBLIGATED minus 
ACTUAL equals Effort Project 

Gap Unplanned 


4. Data Analysis Procedure 

Data analysis should be performed monthly to ensure planned versus actual 
values are within a close range. Actual values indicating inadequate man-hours spent on 
SE activities over a period of time would indicate potential risk of project not having 
enough personnel to fulfill SSC-A SE project requirements. Intervention is recommended 
when actual values decrease from what was originally planned over a period of time. 
Table 9 displays the data analysis procedure for SE staffing and skills. 
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Table 9 

SE Staffing and Skill Data Analysis Procedure 

Indicator 

SE Staffing and Skill Trends 

Frequency of Data 
Reporting 

Monthly 

Responsible 

Individual(s) 

Business Portfolio Manager (BPM) 

SubPortfolio Lead (SPL) 

Integrated Product Team (IPT) Lead 

IPT Technical Lead 

Tier 2 Competency Lead (CL) 

Tier 3 Competency Manager (CM) 

Tier 4 Competency Supervisor 

First-Line Supervisor 

Contracting Officer’s Representative (COR) 

Integrated Product Team 

Activity which 
Analyzed 

Determining the amount of SE being performed in terms of man 
hours. 

Comparing the Supply & Demand of SE / SE Management 
Personnel 

Source of Data for 
Analysis 

NERP, TWMS, Demand Signal Tool, TAA 

Tools Used in 
Analysis 

NERP, TWMS, Demand Signal Tool, TAA 

Review, Report, or 
User 

Review 


5. Historical Data 

Historical SE Staffing and Skills Effort data will be stored in the SEHY database. 
Data elements from NERP such as hours charged for SE efforts will be used to determine 
planned versus actual values. TWMS would provide the number of total personnel with 
specific skills and specialties over time. The number of needed personnel with specific 
skills and specialties would be gathered from the Demand Signal Tool. The TAA 
application would offer data such as personnel with obligated commitments to specific 
project as well as potential availability times. These historical values would produce 
trends for planned versus actual values in regards to SE staffing and skills. 
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6. Sample Visualizations 

SE Staffing and Skill Trends display SE activities performed over time with 
respect to hours of SE effort. Continuous SE throughout the project lifecycle would 
indicate adequate SE practices were being performed. Leadership and stakeholders of a 
project would discuss these trends during regularly scheduled reviews. Figure 23 
displays a notional view of trends for SE staffing. 



Figure 23. SE Staff and Skill Trends (from INCOSE et al. 2010) 
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C. SELI12 PROCESS COMPLIANCE TRENDS 

Process Compliance Trends are used to measure risks of ongoing process 
performance, potential increases in variance, risk to downstream processes, and risks to 
the outcome of the project (after Roedler 2005). As an organization identifies and 
documents its processes, each action is typically included due to its performance of a 
valuable and measurable function output. If the process actions are not completed in 
proper or are neglected completely in performance of the process, then it can be an 
indication that the quality of the overall process and its outputs may not be as desired. 
This reduction of quality can be quantified as a performance risk, which could also 
translate to schedule and cost risks if the resulting products are degraded to the point of 
requiring re-work. Process actions completed out of order or not completed as required 
can be combined and be measured as a general metric covering process discrepancies. 
Measuring process compliance is not the same as measuring process effectiveness. It is 
expected that automating this SELI will be very challenging due to auditing process 
artifacts for compliance and due to the variety of process types within SSC-A. 

Process implementation is the basic requirement necessary before data can be 
collected to show process compliance trends. Industry standards, such as the International 
Organization of Standards (ISO) 9000, identify a process approach as a quality 
management function to be adopted within an effective organization. ISO 9000 identifies 
monitoring and measuring the processes and product against requirements and objectives, 
as well as using the results to continuously improve the processes (ISO 2012). 

Another model, such as Capability Maturity Model Integration (CMMI), 
identified a rating method for an organization’s process maturity. Measuring the maturity 
of processes helps to ensure continued process improvement. This type of measuring is 
not a type of measuring for process compliance but could still be used to show an 
indication of SE health (CMMI Institute 2014). 

SSC-A manages engineering programs and projects using processes in various 
phases of process maturity. Some of the programs have long term customer commitment 
where value is clearly visible for the investment cost of documenting the processes 
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required and managing the compliance discrepancies. Other programs have such unique, 
short-lived functions, that the only documented and repeatable processes are those related 
to the SSC-A business functions. Figure 24 displays the SSC-A business processes 
which are in the construction phase following the guide of the SPAWAR Atlantic Joint 
Framework. 



Figure 24. SSC-A Joint Framework (from SPAWAR Atlantic 2014) 

The intent of this framework is to show all of the business processes, to include 
engineering functions, as a guide to performing work for SSC-A customers. Currently, a 
majority of the high level processes are documented, but all of them are under 
Continuous Process Improvement (CPI). Some of these processes are being delivered 
more like a procedure, which is a difficult challenge that continues to need to be worked 
out as specific process requirements no longer work for all cases. 

SSC-A has already documented a procedure for how to review process 
compliance. This is a procedure that falls under the validation phase of their Process 
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Management process. Figure 25 is a diagram showing the SSC-A Process Management 
process and the validation phase Process Compliance procedure. 



Figure 25. Process Management Procedure Diagram (from SPAWAR Atlantic 
CPI ESG 2014) 
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Figure 26. Process Compliance Procedure Diagram (from SPAWAR Atlantic 
CPI ESG 2014) 


When validating a program or project process, SSC-A shows in steps 7 and 8 of 
Figure 26, the check that the artifact material provided, conforms to the requirement. This 
validation check allows for SSC-A to track the quality of valid process artifacts but not 
specifically process compliance. This type of validation seems to work best for the 
command due to the numerous variations between program and projects, customers and 
sponsors, and industry partners utilized when performing engineering services and 
support. Each project team can provide an acceptable product regardless of the exact 
steps followed within the process. If SSC-A were to build their process compliance 
procedure to capture process variation, then they would also need to measure the 
variation within each IPT to have a more accurate view of program or project risk due to 
inconsistent process compliance. 
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Customers or sponsors desiring to do business with SSC-A may also be interested 
in process compliance trends. If they could also see that the products and services being 
provided were following documented processes and then be able to regularly view an 
updated status of compliance to the applicable processes to their support project, then it 
may increase the reliability they place on SSC-A performance capabilities. 

1. Associated Process 

There are two common variations of process compliance trend measures that an 
organization like SSC-A could use as well as many other individual cases such as the one 
currently possible with the existing SSC-A process compliance validation procedure. 
Each of these use cases would require measuring different data inputs, some requiring 
more or less data than the other, and some having automated tools available to help 
rapidly obtain and measure. 

The first case identified by industry in the SELI guide, is to count the number of 
discrepancies per process required to be followed, and then arrange them in a pareto chart 
showing the highest to lowest discrepancies. This is a chart displaying ordered quantities 
from highest to lowest in each category. Figure 27 is an example of this type of graphical 
representation from the SELI guide. 
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SE Process Compliance Discrepancy Trends 



Analysis: Analysis showed 
that the requirements 
process is not being 
executed... 


Discrepancy 
Expected Process Audit 
Findings Average 



REQ's Risk MSMT 

Process Process Process 

_ PROCESSES Supporting the SEP/SEMP 


Figure 27. SE Process Compliance Discrepancy Trends (from INCOSE et al. 

2010 ) 


The next more detailed use case also identified by the SELI Guide is to arrange 
and graph the identified process discrepancies by the type or by cause of the process 
compliance discrepancy. In this case, it would require the collection of process attributes 
such as process type or discrepancy cause. Figure 28 depicts the SELI Guide example of 
this type of visualization. The number of discrepancies found is shown on the Y axis, and 
the different processes within each process area are listed on the X axis. 
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SE Process Compliance Discrepancies Sorted by Type 





Figure 28. SE Process Compliance Discrepancy Diagram 

Other variations of use cases would include varying the process attribute data 
collected. Some of the attributes helpful to examine for use are quantity of artifacts 
produced, risk or importance of process accuracy, frequency of process use, and time to 
complete process from beginning to end. In the case of SSC-A, as previously shown, they 
will collect the quantity of artifacts produced. 

Tools available for use in measuring process compliance are often limited to the 
tools used to document processes. Configuration Management Professional (CMPro) and 
Atlassian JIRA are two of the known current tools that SSC-A uses to track process 
compliance. On many smaller projects, Excel has been used to list the process steps in 
one column, and then identify a date next to each step when it is completed, or type a 
comment on why it could not be completed if necessary. This method of process 
management is difficult to collect among multiple different projects due to usage of 
different Excel Templates, and minor variations within each IPT’s use of a template that 
may have been provided. 

Data capture from these tools or process audits can occur as often as at the 
completion of every process, or as random as when the next auditing person has time. 
However, industry advice suggests that an organization audit those processes which are 
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most important to success or performed the most often during an identified period of 
time. At SSC-A, a majority of periods of time are based on project schedules, also there 
is very little historical knowledge on process audits since the documented processes are 
still very new and some have not even been documented. Once a process has been 
documented and training provided to users of the process, then it would seem beneficial 
to audit at least one process compliance event per IPT per process area, per funding 
Period of Performance (POP). If the effort is not automated, then it would cost a 
considerable amount of labor in the beginning compared to further in the maturity of the 
organizations use and modification of the processes. 

2, Data Collection Procedure 

Extracting data from CMPro and Atlassian JIRA is required to produce SELI 
trends. Obtaining process compliance data on a monthly basis will ensure up to date 
trends are produced. A historical database is needed to show these trends over time. 
Table 10 displays the data collection procedure for process compliance. 


Table 10. Process Compliance Data Collection Procedure 


Base Measure 

Data Collection Procedure 

Frequency of Data 
Collection 

Varies by process maturity, suggested initially: Once per process 
area, per Period of Performance per project. 

Responsible 

Individual(s) 

Chief Systems Engineer, Process Lead, Quality Assurance 
Manager 

Activity in which 
collected 

All applicable SE Phases 

Potential Sources of 
Data 

CMPro, Atlassian JIRA 

Typical Tools used in 
Data Collection 

CMPro, Atlassian JIRA 

Verification and 
Validation 

Process outcome product quality conformance 

Repository for 

Collected Data 

Suggested: Historical Process Compliance Database 


82 




3. 


Data Storage 


Process compliance trend data should be stored in a historical database. This 
historical data can be pulled and compared to actual current data from a process tool used 
within SSC-A. Since many SSC-A projects use different tools to store their process 
requirements, an interface will need to be able to collect the actual compliance status per 
process, per project, and per portfolio at the frequency interval determined, and store 
within the historical database. 

Table 11 displays data elements that are suggested for collecting and storing 
process compliance data. 


Table 11. Process Compliance Data Elements 


Inputs/Outputs (Data Elements) 
Required 

Tool or Data 

Sources 

Metric/Formula used With 

List of Portfolio Names for Process 
Compliance 

JIRA 

% of Portfolio’s using Process 
Tool 

List of Project Names for Process 
Compliance 

JIRA 

% of Projects using Process Tool 

List of Process Areas per Project, Per 
Portfolio 

JIRA 

Compare to Standard SE Template 
Required for that IPT or Project 
Type 

Count of CURRENT Process Areas 
per Project, per Portfolio 

JIRA 

% of Processes Following per 
Project out of Available defined 
processes in Command 

Count of CURRENT PLANNED 
REMAIN Steps/Artifacts per Process 
Area, per Project, per Portfolio 

JIRA 

Process Compliance Trend 

Count of CURRENT ACTUAL 
REMAIN Steps/Artifacts per Process 
Area, per Project, per Portfolio 

JIRA 

Process Compliance Trend 

Date of Process Data Collection 

Metric Pull 

SEHV 

Database 

Process Compliance Trend 
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Count of PREVIOUS Process Areas 
per Project per Portfolio 

SEHV 

Database 

Process Area Trends 

Count of PREVIOUS PLANNED 
REMAIN Steps/Artifacts per Process 
Area per Project 

SEHV 

Database 

Process Compliance Trend 

Count of PREVIOUS ACTUAL 
REMAIN Steps/Artifacts per Process 
Area 

SEHV 

Database 

Process Compliance Trend 


4. Data Analysis Procedure 

Analysis of process compliance trend data is suggested to follow the existing 
procedure at SSC-A measuring process artifact conformance. This would require 
subtracting the count of artifacts planned to be completed per process area per project, 
from actual current artifacts completed and conforming. A risk could occur here 
depending on how the data is input into the tool(s). If users are expected to self-certify 
the completion of an artifact from a process, then there is no validation of artifact 
conformance. It is suggested to have a separate role audit and certify the artifact as 
complete. Table 12 displays the data analysis procedure for process compliance. 


Table 12. Process Compliance Data Analysis Procedure 


Indicator 

Process Compliance Trends 

Frequency of Data 
Reporting 

As requested by authorized personnel 

Responsible 

Individual(s) 

Command Senior Leadership Staff 

SSC-A Executive Officer 

Code 01B Director of Management Operations 

Code 01E Director of Integration and Efficiency 

Code Oil Inspector General 

Business Portfolio 

Business Portfolio Manager (BPM) 

Deputy Portfolio Manager (DPM) 

Portfolio Systems Engineer (PSE) 
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Integrated Product Team (IPT) Lead 

5.0 Competency 

Tier 1 Local Competency Lead (LCL) 

Lead Systems Engineer (LSE) 

Continuous Process Improvement (CPI) Lead 

Process Owner 

Process Manager 

Integrated Product Team 

Activity which 
Analyzed 

Ensuring process discrepancies on projects are kept low during 
audits 

Source of Data for 
Analysis 

CMPro, Atlassian JIRA, SEHV Database 

Tools Used in 
Analysis 

CMPro, Atlassian JIRA, SEHV Database 

Review, Report, or 
User 

Review 


5. Historical Data 

SSC-A has very limited historical data related to Process Compliance. Any 
current compliance data would need manual input into the SEHV historical database for 
comparison to future collected data; otherwise, initial data collected will need to proceed 
through multiple iterations before enough historical data is collected to form a basis of 
risk to program or project health based on Process Compliance. 

6. Sample Visualizations 

Process Compliance Trends would display a project’s required number of 
processes versus the actual number of processes that the project performed over time. 
Any deviations from expectations would trigger needed improvement efforts. Leadership 
and stakeholders of a project would discuss these trends during regularly scheduled 
reviews. Figure 29 displays an example of trends associated with technical planning. 
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Figure 29. Example Technical Planning Process Compliance Trends 


D. SELI13, TECHNICAL MEASUREMENT TRENDS 

Technical measurement is the method used to track progress of the planned 
Technical Performance Measures (TPMs) against the actual measured value at a 
particular project milestone. The TPMs are identified early in the project life cycle and 
planned values for the TPMs are projected based on the level of information available at 
the time. As the project progresses the planned profile may be updated along with the 
estimated value at completion. The estimated value at completion is then compared to the 
threshold and objective values of the TPM requirement at each milestone review which 
helps to facilitate decision making on how to proceed with the remainder of the project. 
TPMs are typically either derived from Measures of Performance (MOPs) to track system 
mission or functional metrics or derived from Measures of Effectiveness (MOEs) to track 
system cost or effectiveness metrics. 

Technical Measurement involves tracking the following elements: 

• Achieved-to-Date - Measured technical progress of the TPM, or estimate 
of progress, that is captured, recorded, and tracked at the designated 
milestone dates 

• Planned Value - Predicted value of the TPM for the planned date of 
measurement based on the project plan 














• Planned Profile - Series of values representing the projected time-based 
measurement of a TPM requirement describing the expected behavior of 
the TPM over time 

• Tolerance Band - Limits placed on each side of the planned profile to 
indicate the degree of variation accepted based on risk tolerance 

• Threshold - The minimum acceptable value of a TPM requirement 

• Variation (or Demonstrated Technical Variance) - Difference between the 
“Planned Value” and the “Achieved-to-Date” value at a specific point in 
time 

• Current Estimate - “Planned Value” at Estimate of Completion (EOC) of 
the TPM 

• Milestones - Planned and actual dates for measurement, analysis, and 
review 

Figure 30 is a graphical depiction of the Technical Measurement Elements. 


TECHNICAL 

PARAMETER 

VALUE 

e.g., 

MTBF 



TIME 

Figure 3-1 Technical Measurement Profile Illustration 


Figure 30. Technical Measurement Elements (from Roedler and Jones 2005) 
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Associated Process 


TPM is planned early in the life cycle and then performed with increasing levels 
of fidelity as the technical solution is developed. At each milestone review the TPM 
values and evaluated and tracked against the plan. The TPM estimate at completion can 
be recalculated and used to forecast whether the result falls within the acceptable range of 
the tolerance band. Figure 31 shows the Technical Measurement Process. 



Figure 31. Technical Measurement Process (from Roedler and Jones 2005) 


2. Data Collection Procedure 

Extracting data from IBM Rational DOORs is required to produce SELI trends. 
Obtaining technical measurement data on a monthly basis will ensure up to date trends 
are produced. A historical database is needed to show these trends over time. Table 13 
shows the data collection procedure for technical measurement. 





















Table 13. Technical Measurement Data Collection Procedure 


Base Measure 

Data Collection Procedure 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

One Time 

Responsible 

Individual(s) 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 

Base Measure 

(MOE/MOP/TPM) Estimate 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 
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Base Measure 

(MOE/MOP/TPM) Actual 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 

Base Measure 

Maturity State 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 

Base Measure 

Process Phase 
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Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 

Base Measure 

Priority Level 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 

Base Measure 

Date 

Frequency of Data 
Collection 

Varies based upon System Engineering Technical Review (SETR) 
dates 
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Responsible 

Individual 

Lead System Engineer 

Activity in which 
collected 

Technical Risk, Requirements Analysis, Modeling, Design and 
Integration 

Potential Sources 
of Data 

IBM Rational DOORS (Requirements Database); Test Report; 
Requirements Traceability Matrix 

Typical Tools used 
in Data Collection 

Manual Input 

Verification and 
Validation 

N/A 

Repository for 
Collected Data 

SEHV Solution 


3. Data Storage 

Technical Measurement trend data should be stored in a historical database. This 
historical data can then be pulled and compared to actual current data from Technical 
Measurement tools used within SSC-A. Since many SSC-A projects use different tools to 
store their technical measurements; an interface will need to be able to collect the actual 
technical measurements, per project, and per portfolio at the frequency interval 
determined, and store within the historical database. 

Table 14 shows the data elements that are suggested for collecting and storing 
technical measurement data. 
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Table 14. Technical Measurements Data Elements 


Inputs/Outputs (Data Elements) 
Required 

Tool or Data 

Sources 

Metric/Formula used With 

List of Portfolio Names for Technical 

Measurement Trends 

IBM Rational 
- Quality 
Manager 

% of Portfolio’s using Technical 
Measurement Tool 

List of Project Names for Technical 
Measurement Trends 

IBM Rational 
- Quality 
Manager 

% of Projects using Technical 
Measurement Tool 

List of CURRENT PLANNED 
Quality Goals exit criteria, per 
project, per portfolio 

IBM Rational 
- Quality 
Manager 

Trend Counts of Quality Goals, 
per Project/Portfolio 

List of CURRENT ACTUAL 

Quality Goals Test Values, per 
project, per portfolio 

IBM Rational 
- Quality 
Manager 

Quality Goals’ Values Trends 

List of PREVIOUS PLANNED 
Quality Goals exit criteria, per 
project, per portfolio 

SEHV 

Analysis and 
Visualization 

Server 

Technical Measurement Goals 

Trends 

List of PREVIOUS ACTUAL 

Quality Goals Test Values, per 
project, per portfolio 

SEHV 

Analysis and 
Visualization 

Server 

Technical Measurement Goals 

Trends 


4. Data Analysis Procedure 

Data analysis should be performed monthly to ensure planned versus actual 
values are within a close range. Not meeting expected performance measures over a 
period of time would indicate potential risk of inaccurate performance expectations. 
Intervention is recommended when actual values deviate from what was originally 
planned over a period of time. Table 15 shows the data analysis procedure for technical 
performance. 
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Table 15. 

Technical Performance Data Analysis Procedure 

Indicator 

Technical Performance Trends 

Frequency of Data 
Reporting 

Varies based upon System Engineering Technical Review (SETR) 
dates 

Responsible 

Individual(s) 

Code 01B Director of Management Operations 

Business Portfolio 

SubPortfolio Lead (SPL) 

Integrate Product Team (IPT) Lead 

Process Owner 

Process Manager 

Tier 2 Competency Lead (CL) 

First-Line Supervisor 

Contracting Officer’s Representative (COR) 

Integrated Product Team 

Activity which 
Analyzed 

Pre-SETR 

Source of Data for 
Analysis 

SEHV Solution 

Tools Used in 
Analysis 

SEHV Solution; Excel 

Review, Report, or 
User 

Project Manager, 5.0 Leadership 


5. Historical Data 

Historical Technical Measurement trend data may be obtained from IBM Rational 
Quality Manager and stored in the SEHV database. 

6. Sample Visualizations 

Technical Measurement Trends would reveal if parameters are meeting 
performance expectations. Any deviations from expectations would trigger needed 
improvement efforts. Leadership and stakeholders of a project would discuss these 
trends during regularly scheduled reviews. Figure 32 displays a notional example for 
technical performance trends. 
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Technical Performance Index 



Time Measurements 

Figure 32. Technical Performance Index (from Roedler 2005) 

E. SELI16 SYSTEM AFFORDABILITY TRENDS 

Planned cost expenditures along with planned obligation expenditures can be 
compared with actual values to display risks associated with costs over time. Funds used 
within the Government have an expiration date. Different appropriations of government 
funds such as Operations Maintenance (OM), Operations Procurement (OP), Research, 
Development Test and Evaluation (RDT&E) and Base Realignment and Closure (BRAC) 
have different lengths of time for which the funds can be used. An expiring funding 
document overlooked on an AC ATI project can result in significant waste of valuable 
taxpayer dollars. Costs that are not actually expended on time, especially around the end 
of the fiscal year, can notify leadership that potential funds could be pulled to fund 
another critical effort. Risk exposure trends of cost with respect to schedule near the end 
of the fiscal year can provide visibility on a project’s financial plan. Regularly scheduled 
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reviews within a project on this trend should display opportunities for mitigation to 
ensure funds are expended on time and within budget. 

An actual expenditure that is greater than the initially planned expenditure will 
indicate that the project may run out of funds prior to the expected total expenditure date. 
This can cause a serious issue. Personnel being funded by that particular project risk not 
continuing to work to support the effort. Spending too much or too little in regards to 
how severe the deviation from the initial planned expenditures over time can determine 
the severity of the risk. Having visibility of this risk trend will expose a project’s 
financial status and can save the government money by creating a consistent, continuous 
monitoring system that can show these risk trends. If a project has a recurring issue with 
financial management, this can be properly documented with the use of this valued SELL 
Recurring financial problems can alert leadership to intervene with the project’s financial 
situation. Using the System Affordability Trend properly will promote mitigation for 
obvious cost deviations from the original plan. 

System affordability should be determined at several times during the course of a 
project. To properly manage your project each system affordability estimate should be 
compared and contrasted to expected values established from the original baseline 
budget. Figure 33 shows the relationship between project performance, measured in term 
of affordability and associated technical cost and schedule concerns. It also depicts that 
external factors (i.e., union strikes, extreme weather, budget cuts) have strong influence 
over project performance, and they will need to be manage properly for quality project 
execution. 
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Figure 33. Project Performance measured in terms of Affordability (from 
INCOSEetal. 2010) 

In system affordability trends, affordability is defined as the probability or 
confidence of achieving a stated set of needs at a stated cost and schedule. Risk is defined 
at the 100% confidence level to include cost, schedule and performance. When System 
Affordability trends are measured at different project phases (pre-concept, concept, early 
design, design, or implementation), it ensures the leading indicators are performing 
properly. 

1. Associated Process 

If system cost, performance and schedule criteria will be met then the system is 
said to be “affordable.” Affordability is influence by many factors such as: 

• Changes in Stakeholders Requirements 

• System Definition Understanding 

• Interface Demands 

• Technology Maturity 

• Risk Exposure 

• Technical Measures 
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Staffing, Resource Limitations 
Budget Cuts 


All of the above factors can change during a project life cycle, and this will 
impact “Affordability” and its confidence levels. We need to capture the changes of the 
factors and relate them to affordability and confidence levels. Suppose the stakeholders 
wanted to change the requirements; how would this affect cost? At a new time interval 
that represents the changes of requirements, the project manager would determine the 
new cost and associated confidence levels and proposed this information to the 
stakeholders. If the stakeholder wants the same confidence level as before, then cost will 
increase, or if cost is fixed, then the confidence level will be lower. Basically these 
indicators imply that the requirements changes causes more project uncertainty. 
Therefore, the project manager should have conversations with the stakeholders to 
determine if changes in cost and confidence levels are something the stakeholder is 
willing to accept. 

Figure 34 shows another approach to calculate changes in “Affordability.” 
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Figure 34. Affordability Calculation (from INCOSE 2010) 


Changes in affordability can be obtained by adding the sum of the new changes in 
effort to the baseline effort and then dividing this number by the original baseline effort. 

Measurable Concept: Is the SE effort progressing towards a system that is 
affordable for the stakeholders? 





Leading Insight Provided: Indicates whether the merging system is affordable 
and what factors might be driving affordability. 


2. Data Collection Procedure 

Extracting data from NERP is required to produce SELI trends. Obtaining system 
affordability data on a monthly basis will ensure up to date trends are produced. A 
historical database is needed to show these trends over time. Table 16 displays the data 
collection procedure for system affordability. 


Table 16. 

System Affordability Data Collection Procedure 

Base Measure 

• Baseline cost with associated confidence 

• Planned cost with associated confidence 

• Baseline effort with associated confidence 

• Planned effort with associated confidence 

• Baseline schedule with associated confidence 

• Planned schedule with associated confidence 

Frequency of Data 
Collection 

Baseline and monthly reviews 

Responsible 

Individual(s) 

PM and IPT BFM (Business Financial Manager) 

Activity in which 
collected 

Dollars, labor hours, travel and material expenses 

Potential Sources 
of Data 

Project (Schedule), NERP (Cost) 

Typical Tools used 
in Data Collection 

Project (Schedule), NERP (Cost) 

Verification and 
Validation 

Via NERP financial report or PM excel financial spread sheets 

Repository for 
Collected Data 

NERP - Financial Excel master sheet 
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3. 


Data Storage 


The SEHV system will interface and pull data from existing SSC-A system 
engineering and financial tool systems. The SEHV system will store project baseline, 
planned and actual: cost with associated confidence, effort with associated confidence 
and schedule with associated confidence. Cost will be in the form of dollars, effort in the 
form of man hours and schedule in the form of time. Table 17 displays the data elements 
that are suggested for collecting and storing system affordability data. 


Table 17. System Affordability Data Elements 


Inputs/Outputs (Data Elements) 
Required 

Tool or Data 

Sources 

Metric/Formula used With 

List of Portfolio Names for System 
Affordability 

NERP/P2MC 

% of Portfolio’s using 
Affordability Tool 

List of Project Names for System 
Affordability 

NERP/P2MC 

% of Projects using 
Affordability Tool 

CURRENT BASELINED COST per 
Project, Per Portfolio 

NERP/P2MC 

Baseline 

CURRENT BASELINED COST 
CONFIDENCE per Project, Per 
Portfolio 

NERP/P2MC 

Baseline 

CURRENT BASELINED EFFORT 
per Project, Per Portfolio 

NERP 

Baseline 

CURRENT BASELINED EFFORT 
CONFIDENCE per Project, Per 
Portfolio 

NERP 

Baseline 

CURRENT BASELINED SCHEDULE 
per Project, Per Portfolio 

NERP/Project 

Baseline 

CURRENT BASELINED SCHEDULE 
CONFIDENCE per Project, Per 
Portfolio 

NERP/Project 

Baseline 

CURRENT ACTUAL COST per 
Project, Per Portfolio 

NERP 

Actual 


100 




CURRENT ACTUAL COST 
CONFIDENCE per Project, Per 
Portfolio 

NERP 

Actual 

CURRENT ACTUAL EFFORT per 
Project, Per Portfolio 

NERP 

Actual 

CURRENT ACTUAL EFFORT 
CONFIDENCE per Project, Per 
Portfolio 

NERP 

Actual 

CURRENT ACTUAL SCHEDULE 
per Project, Per Portfolio 

NERP/Project 

Actual 

CURRENT ACTUAL SCHEDULE 
CONFIDENCE per Project, Per 
Portfolio 

NERP/Project 

Actual 

PREVIOUS BASELINED COST per 
Project, Per Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS BASELINED COST 
CONFIDENCE per Project, Per 
Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS BASELINED EFFORT 
per Project, Per Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS BASELINED EFFORT 
CONFIDENCE per Project, Per 
Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS BASELINED 

SCHEDULE per Project, Per Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS BASELINED 

SCHEDULE CONFIDENCE per 
Project, Per Portfolio 

SEHV 

Database 

Baseline 

PREVIOUS ACTUAL COST per 
Project, Per Portfolio 

SEHV 

Database 

Actual 

PREVIOUS ACTUAL COST 
CONFIDENCE per Project, Per 
Portfolio 

SEHV 

Database 

Actual 
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PREVIOUS ACTUAL EFFORT per 
Project, Per Portfolio 

SEHV 

Database 

Actual 

PREVIOUS ACTUAL EFFORT 
CONFIDENCE per Project, Per 
Portfolio 

SEHV 

Database 

Actual 

PREVIOUS ACTUAL SCHEDULE 
per Project, Per Portfolio 

SEHV 

Database 

Actual 

PREVIOUS ACTUAL SCHEDULE 
CONFIDENCE per Project, Per 
Portfolio 

SEHV 

Database 

Actual 


4. Data Analysis Procedure 

Data analysis should be performed monthly to ensure planned versus actual 
values are within a close range. Cost increases over a period of time would indicate 
potential risk of inaccurate initial estimates or unforeseen events unearthed from 
inadequate requirements. Intervention is recommended when actual values deviate from 
what was originally planned over a period of time. Table 18 displays the data analysis 
procedure for system affordability. 
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Table 18. System Affordability Data Analysis Procedure 


Indicator 

System Affordability Trends 

Frequency of Data 
Reporting 

Baseline and Monthly 

Responsible 

Individual(s) 

Code 01B Director of Management Operations 

Business Portfolio 

Business Portfolio Manager (BPM) 

Deputy Portfolio Manager (DPM) 

SubPortfolio Lead (SPL) 

Integrated Product Team (IPT) Lead 

5.0 Competency 

Contracting Officer’s Representative (COR) 

Integrated Product Team 

Project Manager 

Activity which 
Analyzed 

NERP reports and Microsoft Schedule 

Source of Data for 
Analysis 

NERP, labor, travel material, ODC 

Tools Elsed in 
Analysis 

NERP 

Review, Report, or 
Elser 

Monthly Report derived from NERP, Excel 


5. Historical Data 

Historical SE system affordability trend data may be obtained from NERP and 
stored in the SEHV database. 

6. Sample Visualizations 

Figure 35 compares a sequence of affordability distributions at various times (Tl, 
T2, T3,) during the course of a project. Initial affordability distribution is set at the 
baseline cost and confidence for the stated requirements. 
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Affordability - Cost Trend 



Affordability - Confidence Trend 



Time 

Figure 35. Affordability Cost / Confidence Trends (from INCOSE et al. 2010) 

Figure 35 represents changes in affordability and confidence level as follows: The 
top chart represents cost trends if confidence is held constant and the second chart 
represents the opposite, the confidence trends if the cost is held constant. When 
comparing the above two charts at T1 the confidence level is high and the cost is low. At 
T2 cost has increased in the project and confidence level is reduced. At T3, the charts 
show lower cost and the confidence level has increased. 
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V. MODELING AND SIMULATION 


Modeling and simulations help portray processes in a virtual environment. 
Predicting potential outcomes from simulations assist with facilitating a way forward for 
decision makers. Gaining insight from simulation outcomes can display current process 
as well as potential areas for improvement. The continuous effort for reducing inefficient 
processes can result in cost saving opportunities, performance enhancements, and time 
reducing options. Optimizing operations by simulating a current process and proposing a 
more efficient solution will communicate the feasibility of a possible future plan. Models 
and simulations show real world activities under different conditions while saving on 
costs by reducing the need to perform activities manually. Refining models to illustrate 
real world scenarios add value to continuous process improvement initiatives. The 
Extendsim simulation contains timing values for 8-hour days, 5 days a week. This would 
align with the standard DOD 40-hour workweek. Process delays were simulated using 
triangular distributions containing minimum, maximum, and most likely values. 

The need for the most up-to-date information is always preferred. The more 
current the data, the earlier leadership could take action on potential risk areas. Consistent 
fresh data would benefit SSC-A by depicting the current health of the organization with 
respect to SE. The current manual data call process usually results with data that is 
slightly outdated. The staleness of the data received by leadership was analyzed during 
the Manual and SEHV simulations. 

A. MANUAL MODEL 

The manual data call process involves multiple personnel to complete the process. 
The start of the manual data call typically comes down from leadership where a specific 
measure is requested. Measures are helpful for a particular instance of time, however, 
they are even more beneficial when being captured consistently, accurately, and 
compared against historical data. The tedious data call process would begin by 
distributing emails requesting various measures. Excel sheets would typically be 
populated with the data requested by leadership and filled out by IPT members. The IPT 
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members would send the measures back to the IPT lead or requestor for review. If the 
data is accepted by the IPT lead or requestor, the data is then sent to the Strategic 
Analysis and Decision Support (SAnDS) team to review and upload to these measures for 
visual trending depictions. If the measures that the IPT member submitted were 
questionable or incorrect, the reviewer would send an email requesting clarification or 
correction. The manual data call simulation displays this concept by simulating a path 
back to the originator of the email. 

The simulation shows personnel taking action on their part of the data call, 
whether it is inputting the data, reviewing the data, or requesting measures. Delay values 
were set for each action taken by personnel to point out areas where potential risk could 
take place in regards to having a successful data call. These simulations do not account 
for personnel who are on leave, sick, or unable to perform the task during the process. A 
hundred data calls were simulated to obtain an average amount of time a data call would 
take to complete. Figure 36 shows the ExtendSim manual data call process model. 
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B. GENERIC SEHV MODEL 


Figure 37 depicts the SEHY model which automates a part of the SE data call 
process by utilizing a methodology entailing data to be captured, stored, analyzed, and 
displayed. The SEHV methodology allows users to perform SE functions using SE tools 
while data captured from these tools would display trends to leadership that indicate 
potential project risk. This alternative to the manual SE data call would gather data 
without being disruptive to personnel. Information requested by leadership (i.e., SE data 
calls) would be available at their convenience by having access to the SEHV display 
capability. As shown in the figure, leadership would have their own process where they 
would access the SEHV database and be able to select the visualization trends of current 
and historical SE data. SE / SE management would perform SE functions on a regular 
basis using SE tools without being asked by leadership to provide SE data. The SEHV 
system would split the typical data call process into two by allowing an automated 
methodology to provide services to leadership and the SE / SE manager by performing 
functions such as capture, store, analyze, and display. 

This simulation assumes tools being used by SE / SE management have the 
capabilities to input the required data elements for visualization. Another assumption 
would include that leadership, and SE / SE managers are trained on the SEHV system and 
understand how to use SE tools associated with data capture by the SEHV system. 
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Figure 37. ExtendSim SEHV Process Model 


C. MANUAL VERSUS SEHV COMPARISONS 

Typical information requested by leadership (i.e., data calls) would disrupt the SE 
/ SE managers, IPT leads, or any IPT member from their usual daily tasks. Coordination 
between leadership and data collectors would have to be established in order to organize 
a strategy to collect data needed. Consistency of the data being provided and data 
requested is essential for having a successful data call. The SEHV system would allow 
SE / SE managers not to be disturbed by data calls, but promote a means of performing 
SE / SE activities within SE tools while data needed would be collected in a non¬ 
disturbing fashion. 

1. Manual Simulation 

Simulating the manual data call process involved displaying the “staleness” 
values over multiple iterations of data calls. The average age, staleness, max age, and min 
age was recorded to display how stale data could be from a manual data call process. 
Considering the 5 day work week, the results indicate that data could be as old as a 
couple of months. Stale data could re-engage leadership in requesting more updated data 
108 







right after a data call has been completed. The average data call would be completed 
within a month. Figure 38 displays the staleness of data from the manual process. 



The processing time was collected to see how much of a delay there was within 
the manual data call process. The IPT lead would have to coordinate a strategy to collect 
the data from each SE / SE manager once notified by leadership. The SE / SE manager 
would react to the data call by acknowledging its existence, and plan to fulfill the data 
call requirement when time permits. Depending on whether the data call request is 
answered correctly, rework for the submitted information would be possible. The IPT 
lead would have to review the data received to ensure the data call is answered 
sufficiently. Passing this data off to the SAnDS team, would require the SAnDS team to 
review and upload the data for visualization purposes. The process for the IPT lead 
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orchestrating the strategy for collecting data from the SE / SE manager(s) and reviewing 
the data for accuracy would take over a week. The SE / SE manager fulfilling the data 
call request would typically be given a week to provide the data requested. This tedious 
process would have an average of a month to complete a successful data call. Figure 39 
displays the results of the manual simulation that performed 100 iterations of the process. 
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Figure 39. Manual Process Times 


Figure 40 displays the distributions for the values, where the standard deviation 
was a few days. The 25% quartile illustrated approximately 11 working days while the 
75% quartile depicted approximately 15 working days. The values distributed clearly 
show that stale data is present at the completion of a data call. 
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Figure 40. Manual Process Distribution in Days 


2. SEHV Simulation Results 

One of the key points for the comparison is the number of people involved in the 
process. The greater the number of people involved in any process creates a more likely 
occurrence of a process delay or even work stoppage. The more people there are within 
SSC-A’s typical data call process, the more risk of an unsuccessful data call occurring. 
People replaced with automated technology can decrease the chances of human error. 
Technology can reduce the dependency of personnel performing specific tasks. 
Maintenance of electronic components is always a necessity; however the right 
redundancy or backup component can help alleviate potential problems. Redundancy 
with human personnel is possible, but having an exact clone that performs exact functions 
is highly unlikely. Limiting the amount of personnel within a data call process will 
decrease the amount of human error, increases the chances of repeatability, increases the 
speed of the process, and promotes a more efficient process. 
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The SEHV solution includes SE / SE manager(s) to perform their SE duties using 
SE software tools. Assuming the SE / SE manager is trained to use the SE software tools, 
the staleness of data would be on average a week old. Consistent data collection by 
automated processes assists with having current “fresh” data. Regular use of SE tools 
would be a requirement to obtain data for visualization. Figure 41 displays the 
ExtendSim SEHV results. 



Figure 41. ExtendSim SEHV Process Staleness 


Process times for SE functions performed were captured to observe averages and 
instances of time spent on SE and leadership activities. The average time for the SEHV 
process resulted in about 45 minutes. SE / SE management would use SE tools while the 
SEHV system would collect and provide the data to leadership for viewing purposes. 
Figure 42 displays process times for leadership (Process Time: L), and SE / SE 
management (Process Time: SE). 
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Figure 42. SEHV Process Times 

The SEHV process times were captured to point out the need for a consistent and 
continuous methodology for data calls. The luxury of performing SE functions while 
allowing leadership to obtain data via the SEHV system will help eliminate inefficient 
processes. The 25% quartile was around 34 minutes while the 75% quartile was close to 
an hour. These values represent the combined effort of both leadership obtaining the 
desired visualization option and SE / SE Management performing necessary SE activities 
with SE software tools. These values were set to minutes since leadership obtaining data 
would be at their own leisure. SE / SE management would regularly update SE tools as 
part of their normal duties. Figure 43 displays SEHV process distributions. 
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Figure 43. SEHV Process Distribution in Minutes 


3. Experiments 

Frequency, of how often the SEHV methodology should be implemented, was 
determined by providing various frequency inputs to obtain results. Different results 
were analyzed to compare the age or “staleness” of the data delivered. Triangular 
distributions were used to have random amounts of days within a range of possible 
frequency inputs. The less frequent SE / SE Management uses SE software tools assisting 
the SEHV effort, the more outdated the data would be once leadership obtains the visual 
trends. This was proven in the SEHV ExtendSim simulation. The right amount of time to 
be able to react to potential project risk at all levels whether its leadership, portfolio, IPT, 
or competency is critical to the decision maker. 
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Using 5-day 40-hour workweeks to replicate current SSC-A work requirements 
would assist the simulation to determine the frequency of SE software tool use. A 
triangular random value distribution was used to simulate data input of SE / SE 
management personnel. Performing SE / SE management functions within SE tools using 
values of range of frequencies from the following: 


Simulation Frequency Experiment 1 
Minimum: 5 days 
Maximum: 15 days 
Most Likely: 10 days 

Simulation Frequency Experiment 2 
Minimum: 10 days 
Maximum: 20 days 
Most Likely: 15 days 

Simulation Frequency Experiment 3 
Minimum: 15 days 
Maximum: 25 days 
Most Likely: 20 days 

Simulation Frequency Experiment 4 
Minimum: 20 days 
Maximum: 30 days 
Most Likely: 25 days 


Weekly (five day) increments were inserted into the simulation to obtain a MS 
Excel spreadsheet of data from 100 iterations during each experiment. Analytical 
software JMP was used to compare the distributions among the simulation runs. 
Reviewing the results and Experiment 3 with data staleness of a mean of 10 days is 
sufficient to observe current, up to date risk trends. Although some projects may use SE / 
SE management tools more frequently, a requirement of inputting data into these tools at 
least once a month by personnel is recommended to have the most up to date 
visualization trends. Figure 44 and Figure 45 display the results of the simulation 
frequency experiments. 
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Figure 44. Simulation Frequency Experiment 1 and 2 
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Figure 45. Simulation Frequency Experiment 3 and 4 


4. Conclusion 

Creating a model, for both the SEHV and manual data call processes, assisted 
with simulation in a virtual environment. Inefficiencies were observed within the manual 
data call process where potential delays would increase the risk of having a timely data 
call. Reducing risk by reducing delays would promote overall efficiency. Automating 
areas within the data call process would save time and money by taking away manual 
human efforts that have possible room for error. Performing SE / SE Management 
functions within SE software tools at least once a month would satisfy the need of having 
consistent, up-to-date visualization trends. 
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VI. SYSTEM ANALYSIS 


The SEHV system, although having many reusable concepts for other engineering 
organizations or enterprises, in this case only applies to the SSC-A application. In that 
sense, this analysis will focus on mostly the Cost, Schedule and Performance within the 
SSC-A boundary. 

A. COST ANALYSIS 

The estimated costs of the SEHV System options were developed and evaluated at 
a high level based on several reasons and assumptions. The team identified each of the 
major cost categories that would require material and/or labor effort to perform and 
estimated rough values for each based on the options capabilities or specific 
requirements. The first option was to continue performing the Systems Engineering data 
calls via manual method. Even though the manual solution has proven to be unable to 
provide SSC-A with a consistent format for visualizing combined results based on 
stakeholder roles, it was still included in the cost analysis for comparison in order to 
show any cost benefits that result in addition to performance increases. The second 
option, the Automated Hybrid option, was selected based on the AHP data determining 
this automated solution to perform better suited to SSC-A stakeholder needs. Each of the 
other Automated procurement solutions would both have similar cost savings; however, 
the assumed development and integration costs for the complete GOTS solution would 
far exceed the COTS or Hybrid options. This assumption could also be re-evaluated 
based on COTS maintenance and licensing fees. We assume that the COTS solution 
would cost less to develop than the Hybrid Solution would; however, estimating those 
costs would only serve to show a greater or similar potential savings than what the 
Hybrid would provide. 

Each of the cost areas for the SEHV solutions was identified based on the system 
functional architecture. The estimated cost of each option was broken out by the systems 
engineering and logistics life-cycle described by the SSC-A Framework. 
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Manual Method Costs 


It is estimated that each project or systems engineer provides data required filling 
at least one data call per project, and that ten projects are managed each year on average 
by each. With approximately 1000 project engineers at SSC-A managing this average 
project load, it is expected that 10000 annual data calls will be requested. Some data calls 
come from Competency Leads, some come from the Portfolio, and some come from 
customers. On average it is expected to take four hours of labor to collect, analyze, and 
provide an approved display of the requested project data. This does not include any 
waiting time. Weekly Action Reports alone are an example of one recurring data call that 
absorbs approximately 30 minutes of consolidated reporting each week from the SE or 
IPT Lead, per project. Some of these identified projects are actually groups of multiple 
similarly tasked projects, each requiring weekly update information. If a data call is 
defined as a higher level request from the customer or command leadership, then it will 
offset the average expected data call labor effort time spent to respond. 

When evaluating the estimated cost for performing manual data calls, both fixed 
and variable costs were identified. For fixed costs, it is assumed that data was collected, 
stored, and analyzed on the project engineers’ computer and monitor. For variable costs, 
we assumed that each time a data call is requested, it takes the project engineer two hours 
to collect the data in the necessary format, and it takes the IPT lead one hour to analyze 
and one hour to develop the display required. This is a total of four estimated hours of 
labor per data call. 

A mathematical function demonstrating this manual cost per data call can be 
shown as Y=$3000 + $400X, where Y is the total cost and X is the number of manual 
data calls, assuming $100 per hour of labor. The $3000 is a fixed representation of any 
material costs necessary to provide those manual data calls. 

The team assumes no further costs added or saved will apply to this function over 
time due to constant changing projects and engineers. 

For 10000 annual data calls, it is estimated that the total cost is close to $4 million 
each year for SSC-A to staff and provide. 
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2. Automated Hybrid Solution Costs 

For the Automated Hybrid Solution, costs were also separated by fixed and 
variable as shown in Figure 47. In addition, SE planning, design, testing, pre-deployment, 
implementation, modification, operation, and maintenance costs were also considered. 
For the first year, the estimated costs minus operation and maintenance were just under 
$1 million. Annual operation and maintenance costs included approximately $400,000.00 
for maintenance, and only about 1.2 hours of operating labor effort are required per data 
call. This reduced labor time per data call can be extracted from the simulation which is 
based on standard consistent and automated process for collecting, analyzing, and 
displaying SE health visualizations. This cost can be represented with a function 
Y=$400,000 + $120X, where Y is the total cost and X is the number of automated data 
calls, assuming $100 per hour of labor. 

The team assumes that cost changes after the first year will be 10% more or less 
due to software license additions or deletions, as well as the removal of the costs to plan, 
design, test, pre-deploy, implement, and modify the Hybrid tools for use. 

Estimating the operation and maintenance cost for 10000 expected data calls at 
SSC-A using the Hybrid Solution method cost $1.6 million. 

Compared to the manual method, this is a savings of $1.4 million for the first year 
and $2.4 million for each year thereafter. 

When comparing this method’s costs to the manual method, the operations and 
maintenance break even cost and number of data calls, when each of the cost equations 
are solved together for X and Y. This results in X=1418 data calls, at a cost of 
approximately Y=$567,000.00. However, since that cost is less than the Hybrid Solution 
annual operation and maintenance cost, we would need to find the minimum number of 
manual data calls required per year that would equal the cost of operating and 
maintaining the Hybrid Solution. By setting the $1.6 million Hybrid O&M cost equal to 
$3000 + $400,000X, it can be found that if 3993 data calls or more are requested per year 
by SSC-A, then it would be an annual cost savings to invest in the Automated Hybrid 
Solution. Figure 46 and 47 displays the spreadsheet cost detail data for each of these two 
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options. The other options were not addressed as the AHP lead to the Automated Hybrid 
Solution. 



Figure 46. Manual Method Costs Details 
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Figure 47. Automated Hybrid COTS Tools with Central Database Costs Details 


B. PERFORMANCE 

SEHV performance measures include consistency, timeliness of data, and 
accuracy of data captured for trending visualization. The performance of the current data 
call process is inefficient and needs the SEHV methodology to assist. Accuracy of data 
would include using the SE software tools to input information with respect to the 
project. This would alleviate potential errors created by the manual method of entering in 
data onto an excel sheet for the intent of uploading the data into visualization software. 

An assumption that SE tools provide the required data elements that SELI trends 
require to produce visualization graphs was acknowledged during the course of this 
Capstone. Another assumption was that the command will enforce the use of the SEHV 
system for projects to input their data into the agreed upon SE software tools. The use of 
these tools would include a frequency of entering in data of at least once a month to 
obtain fresh data. The SEHV methodology is assumed to be accepted and utilized within 
SSC-A. 
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There is a maintenance risk for the SEHV system. Troubleshooting integration 
issues that may arise due to IA patches or software updates, would require subject matter 
experts of the system to be able to mitigate potential problems. A team would have to be 
trained, and possibly be involved with the development of the SEHV system itself in 
order to adequately be able to provide helpdesk support when issues arise. Another risk 
would include projects that deal with sensitive information that cannot expose certain 
SELIs such as technical performance measures, risk exposure, or any other SELI that 
may affect the classified nature of a particular system. Operational availability of the 
SEHV system would also be a risk if redundant components are not put in place to have 
an adequate failover method. Having a backup database and SEHV server would reduce 
the probability of having a catastrophic failure. 

C. POTENTIAL SYSTEM RISKS 

Table 19 displays potential risks for the SEHV system. These potential risks may 
prevent the SEHV system from being successful at SSC-A, and should be considered 
along with details depicted within this report. Each risk listed has a mitigation strategy to 
reduce risks for developing the SEHV system. 

1. Inconsistent use of SE / SE Management tools can create discrepancies 
when data collection occurs. 

Mitigation - Set a minimal operational requirement for SEHV users to 
input data, provide training, and document standard operating procedures. 

Risk Justification - Medium impact with a low probability since planning 
objectives can be put forth prior to SEHV system deployment. 

2. SEHV system development cost may vary due to lack of detailed system 
specifications required. 

Mitigation - Complete a market survey, economic study, and develop a 
prototype prior to implementing the SEHV system deployment effort. 

Risk Justification - Medium impact and low probability since additional 
cost analysis efforts can be performed. 

3. Information assurance risks can affect SEHV system operation, 
availability, and performance. 
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Mitigation - Confirm information assurance guidelines are met to obtain 
accreditation for the SEHV system to operate within the SSC-A network 
infrastructure. 

Risk Justification - Medium impact and low probability since information 
assurance audits can be performed prior to SEHV system deployment to 
meet accreditation standards. 

4. Visualization change requests over time would affect SELI displays. 

Mitigation - SEHV displays should have the capability to learn, adapt or 
be easily modified. 

Risk Justification - Low impact and low probability since the SEHV 
system has not been developed yet. 

5. Tools may produce similar data elements required for SELI trends. 

Mitigation - A tools analysis will have to be performed to determine 
which tool is the most appropriate to provide the needed data elements. A 
prototype can be produced to test operational functionality. 

Risk Justification - Low impact and low probability since a prototype can 
be developed to demonstrate system concepts and compare additional 
software tools for all 18 SELIs. 

6. Tools may be subjected to manipulation where the SE / SE Management 
tools can only process the data entered versus validating the data entered. 

Mitigation - SSC-A will have to perform audits to validate information 
entered within SE / SE Management software tools. 

Risk Justification - Medium impact and medium probability since there 
may be a chance that false data is entered within the SE / SE Management 
software tools. 

7. Software Tools may not be able to provide required data elements needed 
for SEHV trends. 

Risk Mitigation - Verify software tools can provide data elements needed 
for SEHV trends. 

Risk Justification - High impact with a medium probability because 
without the required data elements, SEHV trends will not be produced. 
Software tools identified within this report have the necessary data 
elements for trends; however, other SELIs may require unique data 
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elements that SE / SE management software tools cannot produce. This 
would result in a medium probability. 

8. Data Elements from software tools may have to be translated into a 
compatible format for the SEHY analyzing capability. 

Risk Mitigation - Ensure data elements harvested from the software tools 
are in a format where storage and analytical functions for SEHY trending 
is in a compatible format. 

Risk Justification - High impact with a medium probability due to the 
need for data elements to produce SELI trends. 

9. SE / SE Management COTS Software Tools may not provide data rights 
to distribute or assist with software code for data element capture efforts. 

Risk Mitigation - Contact commercial vendors to obtain information on 
costs, levels of support, and data rights information in regards to data 
elements for the SEHV system. 

Risk Justification - Medium impact with medium probability since SSC-A 
software engineers may be able to extract required data elements from SE 
/ SE Management tools. 

10. Not enough support to assist with Navy owned SE / SE management tools 
to provide data elements needed for SEHV trending displays. 

Risk Mitigation - Identify stakeholders that can provide direction for key 
personnel that can assist with technical support, and provide concurrence 
for the SEHV system effort. 

Risk Justification - Medium impact with medium probability since 
funding may be distributed in-house to support the SEHV system. 

11. Upgrades to SE / SE management software may affect operation of the 
SEHV system. 

Risk Mitigation - Coordinate a process to receive information on 
upcoming updates for software tools to ensure updates to software do not 
affect the SEHV system. Create standard operating procedures for 
software upgrade verification testing. 

Risk Justification - Medium impact with a low probability since 
coordination for upgrade efforts can be defined prior to SEHV system 
development. 
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Table 19. Risk Table 
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VII. CONCLUSIONS AND RECOMMENDATIONS 


The work performed by the SEHV team will assist SSC-A 5.0 System Engineering 
Management in introducing a new automated method to capture the system engineering 
health of a project thus improving system engineering methodologies and efficiencies at 
SSC-A. Additionally, the SEHV team identified future research requirements and 
provided recommendations for SSC-A consideration. 

A. CONCLUSIONS 

SSC-A stakeholders recognize that a organizational SEHV automated capability 
is the way of the future. SEHV automated functionality is a leading edge process that has 
the potential to assist program managers, high level management, and decision-makers 
view a realistic status of a project and offer insight into determining whether projects 
have a high probability of failing. This insight gives the organization the ability to either 
add additional resources or stop spending additional dollars on a system that is being 
managed inefficiently. It can also show strengths and weakness of its existing workforce 
to determine new training workforce requirements. Currently, SSC-A monitors its 
projects via the traditional cost, schedule and performance indicators. It also processes its 
data calls through manual methods. The purpose of this capstone was to determine if an 
automated means of collecting and displaying SE data trends is feasible and effective. An 
architecture, acquisition and implementation strategy was proposed. To accomplish this, 
the SEHV team analyzed stakeholders’ needs and requirements. A literature study was 
performed to gather what types of SELI should be analyzed per stakeholders’ 
requirements. Modeling and simulation was performed to further analysis stakeholders’ 
requirements as the SEHV team developed a model and simulated manual and automated 
data calls via “ExtendSim.” These operational scenario use cases developed the SEHV 
IDEFO documents via Innoslate, which consisted of five functional areas for analysis: 
Provide SEHV, Perform SE Activities, Provide SE Tools Functions, Maintain SEHV and 
Host SEHV. To accomplish the proposed automated architecture, the SEHV team 
performed an AHP to determine the best acquisition procurement strategy for moving 
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forward. From the results of the AHP, cost estimations were developed for the new 
automated architecture solution and compared to the existing manual system along with 
the associated risk analysis. 

B. RECOMMENDATIONS 

From this research process, the SEHV team made the following 
recommendations: 


1. SSC-A should consider cost savings and operational efficiencies of 
switching from a manual architecture to an automated architecture to 
collect project system engineering health data. 

2. SSC-A should consider an SEHY hybrid solution for their acquisition 
strategy for the automated architecture. This will consist of COTS and 
GOTS developed software. There are existing COTS SE software tools 
that can be leveraged as SSC-A will need to develop GOTS software 
solution for interfacing with the COTS tool and SSC-A’s databases and 
government tools. 

3. From the simulations, the SEHV team concluded that the performance of 
the current data call process is inefficient and needs the SEHV 
methodology to assist. Recommend usage of the system to include SE / SE 
management use of SE software tools at least once a month. This would 
satisfy the need for consistent and up to date data. 

4. SSC-A should consider performing the proposed areas of future studies 
before implementing an automated SEHV system at SSC-A. 

C. PROPOSED AREA OF FUTURE STUDIES 

The SEHV team recognizes that there are several areas of future studies that SSC- 
A should consider before implementing the SEHV system into the organization. Due to 
capstone scheduling requirements, the SEHV team had to make several assumptions that 
reduced project scope and added risk considerations. Below is a list of potential areas for 
future study for SSC-A consideration. 


Continue the development of the SELI stakeholders’ visualization 
requirements. SEHV team based stakeholders’ requirements from the 
SSC-A CONOPS and IPT Handbook, which were validated by 5.0 
competency management. These documents may be outdated or too broad 
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to capture all SSC-stakeholders requirements. The SEHV team 
recommends interviewing stakeholders which may provide additional 
requirements that were not mentioned in this document. 

SSC-A should form dedicated working groups with a variety of 
management positions to determine the types of visualization dashboards 
desired by the command to use. 

SSC-A should perform additional research on all leading indicators to 
determine the data elements that is needed by the SEHV system to provide 
the correct data visualization capabilities to the stakeholder. Due to time 
constraints, only five SELIs were selected for detailed analysis. All 18 
SELIs require further research. 

SSC-A should consider performing additional research on existing SE 
tools to determine the interface requirements with the SEHV 
recommended architecture and existing database. 

As with any sensitive data, SSC-A should consider security policies that 
would need to be developed for authorization of storing, using and 
displaying the data. 

SSC-A should perform additional research on life cycle cost. The cost 
numbers, here within, were based off gathering requirements from SSC-A 
operational documents. These requirements will have to be revisited and 
that may affect cost. Also, additional market research is required for 
COTS SE tools. New stakeholders’ requirements need to compare an 
analysis with existing COTS SE tools to determine if possible to use a 
COTS solution or if there is a need to develop GOTS solutions. Further 
studies will have to be put into place to see how much maintenance will be 
needed. 

SSC-A should consider developing quantitative mission of effectiveness 
values and run historical data through the SEHV ExtendSim existing 
models to gather this additional data. Due to time constraints and issues 
with collecting historical project data, the SEHV team could not run 
historical data through these existing models. 
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APPENDIX A. RISK SUPPORT 


A. APPLICATION RISKS 

The application risks are identified in Chapter VI, Section C, of the main body of 
this document. 

B. SUGGESTED METHODS OF RESOLVING RISKS 

Risks related to IT systems or applications were identified and documented based 
on the methodology in NIST SP 800-30, Risk Management Guide for Information 
Technology Systems. Table 20 identifies the Risk Management Roles and 
Responsibilities. 


Table 20. Risk Management Roles and Responsibilities 


Role 

Responsibilities 

Business SME 
(BSME) 

The BSME assisted in identifying and determining the 
context, consequence, impact, timing, and priority of the 
risk. 

Risk Manager or 
Project Manager 
(PM) 

The Risk Manager or PM determined if the risk was 
unique, identified risk interdependencies across the project, 
verified if the risk was internal or external to the project, 
assigned risk classification and tracking numbers, and , 
continuously monitored the project for potential risks. 

Integrated Project 
Team 

The IPT was responsible for identifying the risks, the 
dependencies of the risk within the project, and the context 
and consequence of the risk. In addition, also responsible 
for determining the impact, timing, and priority of the risk 
as well as formulating the risk statements. 

Risk Owner(s) (RO) 

The Risk Owner determined which risks required 
mitigation and contingency plans; generated the risk 
mitigation and contingency strategies. The RO was 
responsible for monitoring, controlling and updating the 
status of the risk throughout this project. 

Other Key 

Stakeholders 

The other stakeholders assisted in identifying and 
determining the context, consequence, impact, timing, and 
priority of the risk. 
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c. 


MITIGATION EFFORTS FOR CAPSTONE PROJECT RISKS 


Tools and Practices 

A risk management log (RML) was maintained by the PM and RMGR and was 
reviewed as a standing agenda item for project team meetings. 

Risk activities were recorded in the SEHV Risk Document located on the SEHY 
google shared drive. 

Risk Closing 

A risk was considered closed when it met the following criteria: 

• Risk was no longer valid 

• Risk Event had occurred 

• Event was no longer considered a risk 

• Risk closed at the direction of the PM 

Lessons Learned 

The lessons learned were captured and recorded in the SEHV risk assessment 
folder located on google docs. Table 21 and 22 displays the program risk rankings. 


Table 21. Program Risk Ranking 


How to Rank Program Risks 


Individually rank each risk identified according to its frequency/likelihood and its 

severity. _ 

Enter information into white cells. Shaded cells will calculate automatically. _ 

Step 1 - Rank the frequency/likelihood of a given risk from 1 to 4 using these criteria: 


4 = Very Likely - Almost certain to occur over the life of the project (or a 10 year 

period—whichever is shorter) _ 

3 = Likely - Probably will occur during a 10-year period _ 

2 = Unlikely - Probably will NOT occur during a 10-year period _ 

1 = Very Unlikely - Almost certain NOT to occur during a 10-year period _ 
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Step 2 - Rank the severity of a given risk from 1 to 4 using these criteria: 


4 = Very High - Would prevent goals and objectives from being achieved 

3 = High - Would cause significant problems or delays in objectives being achieved 

2 = Medium - Would cause relatively minor problems or delays in objectives being 
achieved 

1 = Low - Would probably not affect project implementation 


Step 3 - Compute the average frequency/likelihood and average severity ranking for each 
risk category. 

The risk assessment tool will calculate and display these averages automatically. 


Step 4 - Add the average frequency/likelihood and severity ranking for each risk 
category. 

The risk assessment tool will calculate and display these sums automatically. 


Step 5 - Determine whether each risk category is high, medium, or low according to the 
following thresholds: 


9-12 High Risk (red) - You should have a detailed mitigation action and perhaps 
consider modifying your goals and objectives 

4-8 Medium Risk (yellow) - You should have a clearly defined mitigation action 

1-4 Low Risk (green) - No mitigation action required (or a very basic action if you think 
it is necessary) 


The risk assessment tool will determine and display whether each risk is high, medium or 
low automatically. 

High risks categories will automatically turn cells red, medium risks will automatically 
turn yellow, and low risks will automatically turn green. 


How to develop Risk Mitigation Actions 


For risks that are essentially internal (e.g. Capacity, Leadership, Partners) focus on taking 
action to reduce the risk. 

For risks that are external to the project (e.g. Political, Economic) response will more 
likely be to develop contingency plans and monitor the risks. 


Risk mitigation strategy should be simple, clear and manageable with the resources 
available. 
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Risk 

Potential 
Impact on 
Capstone 
Report 

1/2/3/4 

Potential 
Impact on 
Project/IPR 
Success 

1/2/3/4 

Likelihood of 
Occurrence 

1/2/3/4 

Score 

1 Through 

12 

Ranking 

L/M/H 

Mitigation Plan 

strongly 

recommended for H/ 
H, H/M and M/H 
Possibly even M/M 

1. Develop Chapter 

1 project report 

2 


1 

3 

L 

M/L 

2. Develop IPR 
practice 1 slides 


3 

1 

4 

M 

M/L 

3. Get Approval of 
PMP (High Risk) 


4 

2 

6 

M 

H/M Keep working 
with advisors until 
we get a signed 
document 

4. Get Proposal 
Approved (High Risk) 

4 

4 

2 

10 


H/H/M Keep working 
with advisors until 
we get a signed 
document 

5. Ensure research 
question or 

developed 

2 


1 

3 

L 

M/L 

6. Failure to Ensure 
OV-1 complete 

(DONE) 


1 

1 

2 

L 

L/L Completed 

7. Failure to deliver 
SEHV context 

Diagram 


4 

L 

4 

M 

H/L 

8. Understanding 
stake holders needs 
(base-lining 
Requirements) 

4 

4 

2 

■ 

| 

H/H/M Continue to 
work with 
(SPAWAR) 
stakeholders until 
requirements are 
base lined 

9. Scoping system 
requirements (ensure 
project does not 
become to large) 

4 

4 

3 

■ 

1 

H/H/M Continue to 
work with 
(SPAWAR) 
stakeholders to 
ensure they 
understand the 
scope of this project 

10. Keeping system 
requirements within 
scope (preventing 

Scope creep ensuring 
that one requirement 
does not get to big to 
handle) 

4 

4 

3 


■ 

H/H/M Continue to 
work with 
(SPAWAR) 
stakeholders to 
ensure they 
understand the 
scope of each 
requirement 

11. Ensuring 

requirements are 

understood by 

stakeholder 

3 

3 

1 

7 

M 

H/L Continue to 
work with advisors 
until all stakeholders 
understand scope of 
the project 

12. PMP refinement 

3 

3 

2 

8 

M 

H/M Keep working 
with advisors until 
we get a signed 
document 
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13. Only deliver 
what is in CONOPS 

3 


2 

5 

M 

H/L Keep track what 
will be delivered 

14. WBSw/PMP 


2 

2 

4 

M 

M/L Completed 

15. Update WBSw/ 
PMP if changes occur 


2 

2 

4 

M 

M/M ensure version 
control used in 
Goggle DOCS 

16. SELI version 2 


2 

1 

3 

L 

M/L Completed 

17. Update SELI 
when necessary 


1 

1 

2 

L 

L/L 18 SELIS are 
set, however, need 
to keep track 
through version 
control 

18. Submit 

requirement to NPS 
professor’s 


2 

1 

3 

L 

M/L 

19. Decomposed 
SE processes 


2 

1 

3 

L 

M/L 

20. List of stake 
holders (grouping) 


2 

1 

3 

L 

M/L 

21. PITCO what 
does it actually mean 
is it risk with the 
project of risk with 
cost and 

performance. How do 
we actually determine 
which risk to use 
base on INCOSE 
SELI risk trends 

4 

4 

4 

J 

E 

r 


Table 22. Additional Risks 


Risk Factors 

Low 

Medium 

High 

Mitigation Ideas 

1) Project Size (Duration or 

Effort) 

< 4 Months 

or 

< 1000 
Hours 

4-6 Months 

or 

1000 to 3000 
Hrs 

> 6 Months or 

> 3000 Hours 

• Decomposition 
(Break into smaller 
phases) 

• Phased 
implementation 

2) Project Scope 

Defined 
and Not 
Large or 
Complex 

Somewhat 

Defined/ 

Large/ 

Complex 

Not Defined or 
Large/Complex 

• Decomposition 

• Add another 
analysis phase 

• Detailed 
specifications 

• Early 

prototype/review of 
functionality 

• Add more time 
to the project 
schedule. 

3) Project Decision-Making 

One 

sponsor & 
One 

Sponsoring/ 

Decision 

making 

No Clear 

Sponsor/Decision¬ 

maker 

• Specify 
decision-makers’ 
role in project 
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decision¬ 

maker 

committee 


documentation 

• Add tasks to 
the Project Plan for 
involving decision¬ 
makers and 
managing 
relationship with 
them 

4) Environmental State 
(Software/Hardware/Network) 

Stable / 
Little 
Change 
Required 

Transitional 
with Some 
Changes 

Volatile or Yet to 
be Deployed 

• Additional 
testing, particularly 
stress testing 

• Training on 
new environment 

• Find other 
projects using a 
similar environment 
to compare notes 

• Get a 

prototype deployed 
ASAP on new 
environment 

5) Team’s Experience (with 
environment/project size/scope) 

Extensive 

(2 or more 
similar 
projects) 

Moderate 

(at least one 
similar 
project) 

Limited 

(No similar project 
completed) 

• Additional 
team training 

• Cross-training 

• Consider hiring 
consultant with 
additional 
experience for 
initial period 

6) Impact on other CDL 

Operations 

None or 
Very Little 

Some 

Change 

Extensive 

Changes 

• Additional User 
Training 

• On-site 
assistance for 
cutover period 

• Phased 
cutover 

• Expose key 
stakeholders to 
prototype early 

7) Project Schedule created by: 

Project 
team with 
standard 
estimation 
methods 

Project team 
using some 
rough 
guesses 
based on 
limited info. 

Mandate from an 
external source 

• Supplement 
resources (outside 
consultants, other 
groups) 

• Try to reduce 
scope of 
deliverable. 

• Is there a 
minimum level of 
functionality for the 
mandated delivery 
date? 

• Add time to the 
schedule to allow 
for slippage. 
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L.1 PMP PURPOSE 


1. INTRODUCTION 


This document is the Project Management Plan (PMP) for the System Engineering Health 
and Visualization (SEHV) Capstone Project, Naval Postgraduate School (NPS) 311- 
132W Master of Science in Systems Engineering (MSSE) / Master of Science in 
Engineering Systems (MSES) Program. This SEHV Capstone PMP serves as the top- 
level planning document for the project. All activities and processes performed for the 
SEHV capstone project shall be in accordance with this plan. This PMP also describes 
the SEHV capstone project organization and documents roles and responsibilities. The 
specific objectives for this document include documentation of the tools, techniques, and 
methodologies to be used to support SEHV capstone project management and execution. 
The intended audience of the SEHV Capstone PMP includes the primary stakeholders for 
project execution purposes, NPS advisors, and other stakeholders for content awareness. 


1.2 PROBLEM STATEMENT 

SPAWAR Atlantic (SSC-A)’s 5.0 leadership does not have adequate visibility into the 
Systems Engineering (SE) health of its programs in order to make timely and accurate 
decisions concerning potential program risks and/or possible mitigation strategies. SSC-A 
currently performs manual data calls to obtain information required to assess a project’s 
current state of systems engineering health. SPAWAR leadership also lacks a means to 
measure the quality of SE work for the projects and programs being executed at the 
command. With an ability to access, store, and analyze the right program information, 
leadership could gain insight into potential systems engineering risk areas in order to be 
proactive instead of reactive to issues that lead to program failures or reduced 
capabilities. This capstone team will develop a conceptual design and model of a SEHV 
capability to provide SSC-A leadership with insight into the SE health of its programs. 

1.3 PROJECT BACKGROUND 

SSC-A strives to provide continuous process improvement when providing quality 
systems to stakeholders. The goal to be more proactive with the identification of system 
health indicators, during earlier phases of system development, matured as system 
failures were often realized after the fielding and operational delivery. The savings for 
early identification and monitoring of system health can be realized in the next examples. 
The Defense Information Management Human Resource System (DIMHRS) is a failed 
project that did not have proper insight into SE health issues. Almost 1 billion dollars was 
spent on DIMHRS and still the project failed to deliver the capability promised. The 
deliverables were supposed to be used across all Department of Defense (DOD) branches 
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of service but ended up only being used by the Army and Air Force. The Army and Air 
Force are currently experimenting with some of the units that were delivered. The lack of 
written requirements and configuration management were major contributors to the issues 
with DIMHRS. Leadership of DIMHRS directed engineers, on several occasions, to 
develop the project and forego the documentation until later. Each branch of service 
wanted it tailored to suit their needs which fueled unrealistic schedules and continuous 
requirement changes. The resulting outcome of the DIMHRS project came nowhere near 
the overall expectation. (Ugone 2010) 

SSC-A has developed human resource systems, such as the Navy Standard Integrated 
Personnel System (NSIPS), the Career Information Management System (CIMS), and the 
manpower control system, Total Force and Fleet Manpower System (TFFMPS), which 
span the complete software development life cycle from conception to sustainment. SSC- 
A has also developed the Veteran Affairs’ (VA) Veterans Relationship Management/ 
Customer Relationship Model (VRM/CRM) and a Master Veteran Index (MVI) system 
and has integrated these systems with numerous other legacy VA systems. There are 
several other projects that are in progress that include concept refinement, development, 
integration, and potentially sustainment of software that are within different stages of the 
project life-cycle. The SEHV solution needs to provide the ability for each individual 
project to explain its own unique representative systems engineering status in order for 
automated analysis of project health to be identified and visualized to command 
leadership. 

Utilizing the data housed in SSC-A’s existing System Engineering (SE) toolsets would 
only provide a snapshot of the status of the project at the moment in time when the 
request of the data is conducted. SSC-A does not have visual trending representations of 
measured SE / SE management processes that can provide visibility into expected project 
performance and potential future states. Dynamic data will need to be captured, stored, 
compiled, and analyzed from the workforce in an automatic or non-disturbing fashion to 
display the most current status. The SEHV team will research different visual depiction 
capabilities available to illustrate the measures extracted from projects in order to provide 
SEHV stakeholders indicators of potential project failure. 

The SEHV capability is needed to save time and cost in the effort to work more 
efficiently. Manual data calls, meetings, and other grueling efforts to provide information 
to leadership is an ongoing struggle at SSC-A. Currently leadership manually conducts 
data calls to obtain necessary SE / SE Mgmt. information. The information requested 
trickles down to lower echelons and ultimately to the System Engineer. Although the 
System Engineer performs their daily activities, they have to make time to provide 
information requested by leadership. At times, the System Engineer does not have time to 
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respond in a timely manner. This takes quite a bit of time when there are multiple 
projects from which to collect data. An individual actually has to track the information 
requested from each project, and analyze this information to ensure this is the correct data 
leadership is requesting. Figure 1 shows the SSC-A current state (without SEHV) OV-1 
diagram. 



Figure 1 SSC-A Current State w/o SEHV OV-1 


1.3.1 Literature Review 

Literature review was conducted and promoted the use of International Council on 
Systems Engineering (INCOSE) Systems Engineering Leading Indicators (SELI) Guide 
version 2.0. The team’s initial research consisted of exploring and reviewing different 
organizations’ / institutions’ publications, briefs, specifications and artifacts. The team’s 
initial review provided insight on how SE metrics are currently being utilized and what 
information is being revealed. 

MIT previously conducted research to determine how to use SE metrics to identify 
problem areas and to predict trends. The study breaks down characteristics of what 
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constitutes a leading indicator and its specific usefulness. Objectives and goals were also 
contained within the document. (Rhodes, Yalerdi and Roedler 2008) 

Paul Montgomery and Ron Carlson provided in an NPS literary work on April 30, 2010, 
a discussion about the Applied Leading Indicators (ALI) SE tool. Data analysis features 
were illustrated that Naval Air (NAVAIR) used for their SE process areas. Historical 
metrics were obtained and analyzed to support the goal of predicting future performance. 
(Montgomery and Carlson 2010) 

INCOSE’s technical product identified leading indicators that were associated with 
various parts of the project life cycle, and include work from reputable sources such as 
the Massachusetts Institute of Technology (MIT), Lockheed Martin Corporation, 
INCOSE, the Lean Aerospace Initiative, the Systems Engineering Advancement 
Research Initiative, and Practical Software and Systems Measurements (PSM). Trends 
and various measurements were defined within the technical product to provide examples 
of needed metrics. The ALI tool was briefly discussed and its capabilities/limitations 
were clearly defined. (INCOSE et al. 2007), (INCOSE et al. 2010) 

1.4 PROJECT OBJECTIVES AND PROJECT SCOPE 

The objective of this Capstone Project is to develop the requirements, architecture, data 
strategy, and conceptual model for the SEHY capability. The team will develop SE health 
metrics by analyzing the SEHV requirements and studying SE metrics developed by 
other organizations, such as the INCOSE SELL The team will identify what data and 
information from SSC-A programs is required for the SEHV capability to perform its 
health analysis. The current state of data collection within the organizations (including 
level of automation) will be documented. The information on what is currently being 
collected and how effectively and efficiently the collection is being done will be used to 
analyze the feasibility and costs of any proposed SELI visualization and inform the final 
recommendations outlined in the Capstone final report. The team will analyze 
stakeholder needs which will drive requirements and capability development. The team 
will also determine what types of information need to be visualized to best meet 
stakeholder needs. 

The SEHV capability context diagram illustrates a simplified diagram of the proposed 
SEHV capability. Projects would be the source for the data elements for both the SE / SE 
Mgmt. data elements. SE / SE Mgmt. tools or other sources will be used to generate these 
data elements. The SEHV capability that is housed within the DC/ESE will use the 
concepts of the INCOSE SELI Guide v2.0 to display views to SEHV stakeholders. The 
stakeholders would be able to review these SELI trends from data collected from projects 
and provide feedback. Figure 2 shows the SEHV Capability Context Diagram. 
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Figure 3 shows the OV-1 which illustrates the SEHV capability. The data source will be 
produced by the system engineer allocated to a project. The system engineer will perform 
SE / SE Mgmt. activities using tools and other sources. The data elements from these 
tools / other sources will be harvested or collected by non-disruptive means to be kept in 
the data repository located within the Data Center / Enterprise Services Environment 
(DC/ESE) at SSC-A. The VBMS project has a Visual Depiction Capability and Data 
Repository within the DC/ESE which can be leveraged for the SEHV capability. This 
capstone project will research means to utilize these capabilities to the fullest when 
introducing the SEHV capability. An SELI analyzing capability will have to be 
introduced to work with the data repository in order to produce data schemas for the 
visual depiction capability. The visual depiction capability will provide displays of SE 
health to SEHV stakeholders in order to initiate and facilitate feedback to the system 
engineer of a project. 
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Feedback 



Figure 3 SEHV Capability OV-1 


Key SE health metrics provide insight into the quality and effectiveness of SE activities 
which will support command objectives of consistent solution delivery and customer 
satisfaction. Functional and physical architectures will support a depiction of the 
collection, evaluation, and dissemination of relevant dynamic data elements that support 
the validated SELI measures. This architecture must also support temporal and pattern 
analysis of the data elements. A method to ingest the identified data elements into the 
SEHV data repository, in an automated manner, will also need to be included. The 
frequency with which the data is pulled from the various SE toolset data sources and 
ingested into the SE health data repository will have to be determined. The combinations 
of data elements and collection processes will be evaluated by modeling and simulating 
the proposed SEHV architecture to identify patterns and trends from historical project 
data and determine how the solution supports the prediction of good and bad project 
outcomes based on SELI established criteria. The output of the project should provide a 
package of technical work which will clearly characterize and describe elements 
necessary to realize an automated SEHV capability. Characterizing SSC-A key 
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stakeholders, prioritizing and analyzing user requirements, exploring feasibility and unity 
of SELI measures, developing architecture, and compiling a description of required 
components will be within the scope of this project. The SEHV data strategy, data 
repository requirements, and technical requirements will be developed to characterize the 
desired SEHV capability. Cost analysis will be performed to determine affordability as a 
factor to the overall SEHV solution. 
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2. PROJECT ORGANIZATION & MANAGEMENT 


2.1 TEAM ROLES & RESPONSIBILITIES 


Figure 4 provides the high-level organizational structure chart for the SEHV Capstone 
Project. 



The SEHV team will be organized according to key activities necessary for completion of 
the project. The team will have a primary and alternate to support these key activities. 
The key activities for this project are project planning, requirements development / 
validation, mapping of tools, requirements / metrics analysis, cost estimation, process 
modeling, risk assessment, identification of use case scenarios, and an analysis of 
proposed metrics and process sets. The capstone advisors will review the scope, 
organization, quality, schedule of the project and provide feedback on a regular basis. 
The team will engage with the stakeholders to identify use cases, requirements and their 
priorities. Table 1 lists the SEHV team roles and responsibilities and identifies each team 
member(s) assigned. 


151 



Table 1 SEHV Capstone Project Team Roles and Responsibilities 


Team Member 

Roles and Responsibilities 

Project Manager and Projec 
Technical Risk Analyst - 

Primary-Clive Sugama 
Altemate-Chester Alonzo 

The Project Manager (PM) role is responsible for the 
planning, monitoring, and controlling activities of the 
project. The PM will take the lead on developing the 
PMP. The PM will also take on the duties for Project 
/ Technical Risk Analyst and Configuration Manager 
(CM). The PM will also assist with identifying key 
stakeholders and will facilitate discussions with 
stakeholders for the purpose of capturing 
requirements, use cases, and metrics needs. 

Lead Systems Engineer (LSE) 

Primary-Chester Alonzo 
Altemate-Clive Sugama 

The LSE will take the lead on the elicitation, 
documentation, and organization of the stakeholder 
requirements. The LSE will capture system use cases 
from the various stakeholders. The LSE will also 
track and manage the interfaces for the architecture. 
Finally, the LSE will track and manage the activities 
performing the verification and validation of the 
architecture. 

Data Architecture Lead - 

Primary-Michael Besco 
Altemate-Theresa Inman 

The Data Architecture Lead will take the lead on the 
architecture for the system. The data architect will 
take the lead in identifying leading indicators that 
would indicate the health of system engineering 
efforts within SSC-A. Additionally, the data architect 
will develop the entity-relationship architecture from 
the SE tools that will map data to the leading 
indicators. 

Software Analyst - 

Primary-Theresa Inman 
Altemate-Michael Besco 

The Software Analyst will analyze the 
implementation of the architecture. The Software 
Analyst will interface with the Architecture Lead 
from SSC-A Enterprise SE Tool Suite group to 
determine the information needed for the architecture 
from the selected tool suite and recommend how to 
automatically collect it. . 

Data Editor - 

Primary-Regina McNeil 
Alternate- Michael Jourdain 

The Data Editor will ensure all documentation is 
formatted, consistent, and contain information 
pertinent to that deliverable. The Data Editor will 
enforce the standards set by the NPS Thesis 
Processing Office on the SEHV project report. 
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Cost Estimator - 

Primary-Michael Jourdain 
Altemate-Regina McNeil 


The Cost Estimator will provide cost and benefit 
analysis on the SEHV capability recommendation. 
The Cost Estimator will also analyze costs associated 
with implementation of metric data collection and 
provide cost projections associated with SEHV 
development. 


2.2 STAKEHOLDERS 

Table 2 Stakeholder Roles and Responsibilities 


Stakeholder 

Roles and Responsibilities 

Advisors - 

Professor Young / Profes: 
Gehris 

The advisors will review deliverables and documents to 
ensure it meets both writing and content related NPS 
standards before they are passed onto the department for 
concurrence. The advisors will guide the SEHV project 
team on the right path to a successful capstone project. 
Assistance with project material and mentoring will be 
provided. Clarifying questions asked by the SEHV team 
will be addressed by the advisors. Advisors will also act as 
a liaison between the students and the SE Department. 

SSC-A Active Stakeholders ■ 

The SSC-A active stakeholders will provide appropriate 
resources to support the SEHV project. The SSC-A active 
stakeholders will also participate in SEHV project meetings 
to be informed about project progress, status and risks. 

SSC-A Projects Group 
(Passive Stakeholders) 
Information Producing Grou 

The SSC-A Projects Group (passive stakeholders), within 
the command, will generate data automatically or through 
non-disruptive means to the SEHV project solution. Data 
elements will be stored and displayed for trend analysis. 
SSC-A projects will be able to assess their SE health from 
the resulting display. 

SSC-A Leadership 
(Passive Stakeholders) 
Information Consumer Grou 

The SSC-A Leadership (passive stakeholders) within the 
command will view trending information from the SEHV 
project solution. The stored data will be displayed for trend 
analysis. SSC-A Leadership will be able to assess and 
provide feedback to projects in regards to SE health. 


2.3 PROJECT MANAGEMENT 
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This PMP will be used to manage project effort, schedules, risks, and other relative 
project elements. All SEHV Capstone Project documents will be maintained in Google 
Docs, and will be used to provide configuration management and version control. SEHV 
Capstone Project Close-Out will include finalizing the Capstone Project final report prior 
to the end of the final NPS quarter. A final IPR will be conducted with stakeholders and 
NPS Advisors (and other NPS Faculty, as appropriate) to support the SEHV Capstone 
Project close-out. 

2.3.1 Project Monitoring & Control 

The SEHV Capstone PM will ensure the SEHV Capstone Project’s progress meets the 
required due dates. Appropriate actions can be taken when the project’s performance 
deviates significantly from the plan. The SEHV Capstone PM will monitor the project 
against the plan by keeping track of project milestones. Project risk will also be 
monitored to identify potential road-blocks during the capstone project time frame. 
Stakeholder requirements will be mapped within the Innoslate tool for requirements 
analysis. Progress reviews will take place during the time-slotted IPRs. 

2.3.2 Requirements Management 

The SEHV Capstone PM will manage the requirements of the SEHV Capstone Project 
products and product components, and will ensure alignment between requirements, 
SEHV Capstone Project plans, and work products. In managing these requirements, the 
PM will ensure documenting SEHV Capstone Project processes for interpretation, 
agreement, and commitment to functional requirements. The SEHV Capstone Project 
requirements will be traceable within Innoslate. Requirements documentation, 
traceability, and addressing changes will take place within Innoslate. IPRs, and other 
mechanisms necessary to ensure that inconsistencies between requirements, project plans, 
and work products are identified, tracked, and resolved, will be conducted. 

Requirements are documented in Innoslate to facilitate tracking and change impact 
analysis. As requirements evolve, traceability is maintained. Requirements are reviewed 
internally during weekly meetings, where inconsistencies and issues are identified to be 
either resolved or escalated for resolution. The requirements document identifies when 
requirements are formally base lined and subject to formal configuration control. Prior to 
that point, the SEHV Capstone PM shall set and maintain informal requirements 
configuration controls within Innoslate. Requested changes to requirements are 
documented and reviewed for technical and management impacts. Approved changes are 
implemented and tracked after the SEHV team and advisors agree on a decision. 
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2.3.3 Schedule Management 


The SEHV Capstone Project schedule baseline is captured in Table 5. Specific 
deliverables will be assessed at the end of each NPS Quarter by the NPS instructors. 
Actions will be taken on major deviations from the baseline schedule in order to meet 
specified schedule and performance measures. Scheduled baseline changes will be 
documented and retained for the life of the project. 

2.3.4 Quality Management 

Quality management will be adhered by the advisors reviewing the SEHV work products. 
IPRs, and any other beneficial reviews deemed necessary, take place to assess the overall 
quality of the SEHV project. The SEHV team will make adjustments, as required after 
each review, to satisfy NPS requirements and meet project milestones. 

2.3.5 Communications Management 

The SEHV Capstone Project requires continuous, predictable communications. Table 3 
show the SEHV communication protocol. 


Table 3 Communications Management 


Audience 

Media 

Purpose 

Topics 

Frequency 

SEHV Capstone Project 
Team 

DCO, L 

Meetings 

SEHV Team Advis 
Comment 

Adjudication i 

Strategy Sessions 

Advisors 

Comments, 

Updates, 

Brainstorming 

Weekly 

(Tuesday) 

SEHV Capstone Project 
Team, SSC-A Active 
Stakeholders 

DCO, Email 

SEHV Capsti 

Project SSC 

Atlantic Meeting 

POA&M, 
Action Items 

Weekly 

(Friday) 

SEHV Capstone Project 
Team, NPS Advisors 

Blackboard, 

Email 

Project Reviews 

Progress, 
Findings, 
Potential Issue: 

Weekly 

(Friday) 

Quarterly 

SEHV Capstone Project 
Team, SE Process 

Subject Matter Experts 
(SME) 

Live Meetin 
Email 

SE Leading Indicate 
Discussion 

SE Proc 

Measures, 

Outcomes 

Weekly 

SEHV Team, 

Live Meetin 

Visualization 

Visualization 

Weekly 
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Visualization SME’s 

Email 

Strategy 

Alternatives, 





Methodology 



2.3.6 Risk Management 

There are two kinds of risks associated with the SEHV Capstone: technical (risks to the 
proposed solutions) and project risks (risks to capstone project completion). These risks 
will be tracked separately. Technical risks are documented (and may be revised as new 
information arises) and will appear in the final report. Project risk will follow the SE 
approach described in Section 3 and will not appear in the final report. 

Risk Management will be executed throughout the life of the SEHV Capstone Project. 
This is a critical step in clearly defining risk thresholds. Risks will be managed using the 
risk tracker within Google Docs. SEHV Capstone active stakeholders, the SEHV team, 
and the NPS advisors will review risks as they arise in regards to schedule and 
performance, and then develop the appropriate mitigation strategy and activities to reduce 
and/or eliminate these risks. Risk reviews will be conducted on a weekly or monthly 
basis, depending on the severity and impact of identified risks. 

2.3.7 Configuration Management 

The SEHV Capstone PM, and the Configuration Manager (CMgr), will ensure CM is 
executed throughout the life of the SEHV Capstone Project. CM will include the use of 
Google Docs to ensure team collaboration on SEHV capstone project documentation. 
Google Docs will serve as the repository for the SEHV capstone project and provide the 
capability to audit document modifications and team member ownership of the change. 
Document review will be conducted via email and/or Sakai between the SEHV team and 
NPS advisors. 
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3. SYSTEMS ENGINEERING APPROACH 


The SEHV team will analyze SE leading indicators to facilitate project risk identification. 
The team will study the INCOSE SELI Guide version 2.0. This guide will be used as a 
control for this project. This guide identifies 18 leading indicators that provide consistent 
insight into future project performance. The use of the SELIs will ensure levels of quality 
for the solution. SELIs will be reviewed and assigned priorities based on their impact to 
project cost and schedule, and system performance as well as their ability to answer 
common questions by engineering managers concerning system risks. Table 4 shows the 
SELI measures that were obtained for trends. 


Table 4 SELI Trends 


SELI 1. Requirements Trends 

SELI 2. System Definition Change Backlog Trends 

SELI 3. Interface Trends 

SELI 4. Requirements Validation Trends 

SELI 5. Requirements Verification Trends 

SELI 6. Work Product Approval Trends 

SELI 7. Review Action Closure Trends 

SELI 8. Technology Maturity Closure Trends 

SELI 9. Risk Exposure Trends 

SELI 10. Risk Treatment Trends 

SELI 11. Systems Engineering Staffing and Skills Trends 

SELI 12. Process Compliance Trends 

SELI 13. Technical Measurement Trends 

SELI 14. Facility and Equipment Availability 

SELI 15. Defect and Error Trends 

SELI 16. System Affordability Trends 

SELI 17. Architecture Trends 

SELI 18. Schedule & Cost Pressure 


The SEHV team will down select SELI based on criteria such as SELI mapped to 
multiple stakeholders, automated method of capturing information, tools that produce 
data elements that are needed from SELI, existing / available SE / SE MGT tools, and 
the cost of tools / software. The data collection process of the SELI choices will provide 
an automated solution that harvests SELI measures needed to indicate project risk areas 
that lead to project failure. Once data is collected display mechanism options will need to 
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be researched to be able to display trends of SELI measures. Figure 5 shows the SEHV 
project approach. 



Final SEHV Recommendation 

Figure 5 SEHV Project Approach 


The SEHV team will utilize an SE approach derived from the IEEE 1220 2005 Systems 
Engineering Process. This process is appropriate to use for this project since this 
approach produces documents / models vice hardware or software. Using this process 
will help facilitate the project’s expected process environment, interfaces, work products, 
and necessary SE tools needed for the SEHV capability. This process will begin by 
obtaining SSC-A stakeholder inputs, then going through the SEHV SE process, and 
ultimately ending up with a recommendation (based on weighted criterion) of SELI 
choices that describe data measure collection methodologies with their respective display 
mechanisms. Figure 6 shows the SEHV SE process. 
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System Engineering 
Health & Visualization 
Capability 

Figure 6 SEHV Systems Engineering Process 


The SEHV team will conduct a literature review to research how other organizations 
perform SEHV processes, including which SELIs and supporting metrics were used. The 
team will perform a requirements analysis to establish a set of requirements for the SEHV 
system and will define operational scenarios. The team will develop functional and 
physical architectures that will support the operational scenarios to evaluate alternatives 
through modeling and simulation. The analyses results will form the basis of the team’s 
recommendation to stakeholders. 

The team will provide requirements and concept of operation information that will be 
included in the final report. The team will also identify and address risks to 
implementation, define mission success requirements, develop preliminary architectures, 
develop use cases, perform model-based systems engineering, process analyses, and 
communicate status to project sponsors. An investigation of current tools, application 
effectiveness, gaps within current tools, and potential trade space between collection 
efforts, cost, and value provided to the user will also take place. Figure 7 shows SEHV 
process diagram where the SEHV team will implement. 
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SEHV Process Diagram 



Figure 7 SEHV Process Diagram 

3.1 PROBLEM DEFINITION/NEEDS ANALYSIS 

The team will define the problem and perform needs analysis for the SEHV capability. 
This will include the stakeholder needs analysis, analysis of SSC-A current capabilities, 
and providing examples of programs that have failed or had issues as a result of a lack of 
insight into their SE health. Analysis of the SEHV capability needs will determine the 
overall boundaries, limitations, and constraints. 

SSC-A is highly in need of a comprehensive understanding through dynamic displays of 
the overall health of its SE process areas against prescribed performance measures. SSC- 
A has the need to access specific SELIs to provide decision-makers dynamic and current 
visibility into quality and effectiveness of SE / SE management activities. The capability 
will assist SSC-A in the identification and diagnosis of potential SE / SE management 
problems either before they occur in order to mitigate them or after they occur in order to 
prevent like problems in the future. The entry criteria will include SSC-A stakeholders 
inputs, and the exit criteria would include documented stakeholder needs. 

3.2 REQUIREMENTS ANALYSIS/METRICS ANALYSIS 

The team will perform requirements analysis that will result in a functional requirements 
baseline for the solution. Requirement analysis will begin with ensuring that the SSC-A 
stakeholder’s needs are adequately documented, understood, and suitable for refinement. 
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These will be reviewed against the project criteria, and agreed upon by the SSC-A active 
stakeholders, NPS advisors, and the SEHV Capstone Project team. The SEHV team will 
analyze SEHV leading indicators to facilitate project risk identification. The team will 
study the International Council on Systems Engineering’s (INCOSE) Systems 
Engineering Leading Indicators (SELI) Guide version 2.0, which will be used as a 
reference for this project. The INCOSE SELI guide identifies 18 leading indicators that 
have been proven to provide insight into future project performance. The use of the 
SELIs will ensure levels of quality for the solution. SELIs will be reviewed and 
prioritized based on their impact to project cost and schedule, system performance, as 
well as, their ability to answer common questions posed by engineering managers 
concerning system risks. The entry criteria includes the project need statement and the 
exit criteria includes a stakeholder analysis, documented SEHV solution functional 
requirements, and a prioritized listing of SELIs based upon SSC-A stakeholder needs. 


3.3 FUNCTIONAL ANALYSIS 

The team will determine the data collection process required for each prioritized SELI, 
along with its respective data mapping, data translation, data storage, and data analysis 
methods. The details of the data elements, data sources, and anticipated positive or 
negative trending patterns will feed into the functions that comprise the SEHV system 
architecture. The data capture, data storage, and data management strategies for the data 
elements known to provide insight into targeted SELI trends - based upon INCOSE’s SE 
Leading Indicator study - will be the outcome of this analysis. The SEHV team will 
develop an IDEFO diagram as a part of the functional analysis. The entry criteria will 
include the data elements, data sources, and notional trending patterns of the most valued 
SELIs and the exit criteria will include an IDEFO diagram depicting data capture methods 
and triggers, data transformation and storage functions, as well as, data management and 
archiving strategies. 


3.4 SYSTEM ARCHITECTURAL MODELING 

The project team will continue to refine the SEHV Context Diagram, as well as, the SSC 
Current State OV-1, and SEHV Capability OV-1 diagrams. The team will develop an 
architecture model to be used to facilitate development of the SEHV capability. 
Functional and physical architecture models will be created to assist in implementing the 
SEHV capability. The functional architecture model will be solution neutral while the 
physical architecture model will contain components that will be evaluated against 
specific criteria such as cost, availability, and feasibility. The team will also develop an 
Entity Relationship Diagram (ERD) to define the physical data model of the prospective 
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relational database structure. The entry criterion includes SEHV IDEFO diagrams, 
positive, negative, and neutral SELI trending patterns, and data elements with associated 
source information. The exit criteria includes a functional architecture that allocates 
SEHV functions to specific components, as well as, an ERD that facilitates the storage 
and analysis of the SEHV dataset. 

3.5 SYSTEM ANALYSIS 

The team will perform an assessment of the SEHV capability against the functional 
requirements and operational use cases. Historical project data will be mapped to the 
proposed entity relationship model and the team will analyze the SEHV capability in 
identifying trends and insight into SE health. Notional graphs and dashboards will be 
created to present the information captured within the SEHV solution to the stakeholders. 
The team will perform a high level cost analysis of implementing the system and a risk 
analysis to identify any risks that SSC-A may encounter in implementing this SEHV 
capability. The entry criteria would include the functional architecture and physical 
architecture and the exit criteria will be a validated architecture against historical data and 
operational use cases along with potential visualization mechanism s . 
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4. PROJECT MILESTONES & DELIVERABLES 


Deliverables will include a proposal, three IPR’s and a final report. Table 5 lists the 
POA&M. 

4.1 SCHEDULE 


Table 5 SEHVProject POA&M 


Task Name 

Duration 

Start 

Finish 

SSC LANT System Engineering Health & 
Visualization 

190 days 

Fri 3/21/14 

Fri 12/12/14 

NPS 1st Quarter 

61 days 

Fri 3/21/14 

Fri 6/13/14 

Strategy and Rollout 

36 days 

Fri 3/21/14 

Fri 5/9/14 

Draft Proposal 

6 days 

Fri 3/21/14 

Fri 3/28/14 

Metrics Gathering Strategy 

14 days 

Fri 3/21/14 

Wed 4/9/14 

Final Capstone Proposal (Due to the Advisor Inbox 
in Resources) 

1 day 

Fri 4/11/14 

Fri 4/11/14 

PMP Draft Submission 

1 day 

Fri 4/25/14 

Fri 4/25/14 

PMP Final Submission 

1 day 

Fri 5/9/14 

Fri 5/9/14 

Problem Definition / Needs Analysis 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Document and express SSC-A Active stakeholder 
needs 

16 days 

Fri 5/9/14 

Fri 5/30/14 

Define the problem definition 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Produce a requirements document 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Requirements Analysis / Metrics Analysis 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Identify Top Level “Driving Requirements” 

10 days 

Fri 5/9/14 

Thu 5/22/14 

Stakeholder Analysis 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Analyze organizational SE & SE Mgmt. 
stakeholders and characterize their SELI needs 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Data Source Identification 

20 days 

Fri 5/9/14 

Thu 6/5/14 
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Analyze data elements produced by SELI 

20 days 

Fri 5/9/14 

Thu 6/5/14 

Chapter 1 of the Capstone Report 

14 days 

Tue 5/20/14 

Fri 6/6/14 

IPR Dry Run 

11 days 

Fri 5/23/14 

Fri 6/6/14 

IPR 1 

5 days 

Mon 6/9/14 

Fri 6/13/14 

End of 1st Quarter Stakeholder Approval 
(Control Gate) 

0 days 

Fri 6/13/14 

Fri 6/13/14 

NPS 2nd Quarter 

65 days 

Mon 6/16/14 

Fri 9/12/14 

Requirements Analysis / Metrics Analysis 
(Cont’d) 

38 days 

Mon 6/16/14 

Wed 8/6/14 

Most valued SELI based on the down select 
process 

20 days 

Mon 6/16/14 

Fri 7/11/14 

Down select SELI based on criteria such as SELI 
mapped to multiple stakeholders, automated 
method of capturing information, tools that 
produce data elements that are needed from 
SELI, existing / available SE / SE MGT tools, 
and cost 

20 days 

Mon 6/16/14 

Fri 7/11/14 

Document SEHV solution functional requirements 

10 days 

Thu 7/24/14 

Wed 8/6/14 

Document system concepts for the necessary 
automated data collection procedure 

10 days 

Thu 7/24/14 

Wed 8/6/14 

Functional Analysis 

19 days 

Wed 8/6/14 

Mon 9/1/14 

Develop a SEHV IDEFO 

19 days 

Wed 8/6/14 

Mon 9/1/14 

Identify and describe any additional information, 
mechanisms or integration needed to provide 
the SEHV solution 

10 days 

Wed 8/6/14 

Tue 8/19/14 

Identify system concepts to pull / push data onto 
the data model from sources 

10 days 

Wed 8/6/14 

Tue 8/19/14 

Identify system concepts for all data elements 
needed 

10 days 

Wed 8/6/14 

Tue 8/19/14 

Identify system concepts for data storage 

10 days 

Wed 8/6/14 

Tue 8/19/14 

Determine Display Mechanism Options to generate 
trend analysis 

10 days 

Tue 8/19/14 

Mon 9/1/14 

Assess available visualization tools or software 
components that can be used to generate 
graphical depictions of SELI in INCOSE SELI 
v2 specification 

10 days 

Tue 8/19/14 

Mon 9/1/14 
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Chapter 2 & 3 of the Capstone Report 

21 days 

Fri 6/27/14 

Fri 7/25/14 

IPR 2 Draft 

16 days 

Fri 8/1/14 

Fri 8/22/14 

IPR 2 Rehearsal (advisors only) 

10 days 

Mon 8/25/14 

Fri 9/5/14 

Chapter 4 & 5 of the Capstone Report 

35 days 

Mon 7/28/14 

Fri 9/12/14 

IPR2 Presentation (All invited Stakeholders and 
Faculty) 

18 days 

Wed 8/20/14 

Fri 9/12/14 

End of 2nd Quarter Stakeholder Approval 
(Control Gate) 

Odays 

Fri 9/12/14 

Fri 9/12/14 

NPS 3rd Quarter 

64 days 

Mon 9/15/14 

Fri 12/12/14 

System Architectural Modeling 

10 days 

Mon 9/15/14 

Fri 9/26/14 

Operational use cases of SEHV capability 

10 days 

Mon 9/15/14 

Fri 9/26/14 

Assess available SE tools and other sources 
reporting and query mechanisms to identify 
how required data elements can be provided for 
SELI use 

10 days 

Mon 9/15/14 

Fri 9/26/14 

Develop a Entity Relationship Diagram for display 
mechanisms 

10 days 

Mon 9/15/14 

Fri 9/26/14 

Refine Diagrams (OV-1 & Concept) 

10 days 

Mon 9/15/14 

Fri 9/26/14 

System Analysis 

20 days 

Mon 9/15/14 

Fri 10/10/14 

Perform a high level cost analysis of implementing 
the system 

20 days 

Mon 9/15/14 

Fri 10/10/14 

Risk analysis to identify any risks that SSC-A may 
encounter in implementing the SEHV 
capability 

20 days 

Mon 9/15/14 

Fri 10/10/14 

Validate architecture against historical data and 
operational use cases along with potential 
visualization mechanisms 

10 days 

Mon 9/15/14 

Fri 9/26/14 

Draft Capstone Project Report 

18 days 

Fri 10/3/14 

Tue 10/28/ 
14 

Draft Capstone Project Report (NPS Req.) 

46 days 

Fri 10/3/14 

Fri 12/5/14 

Use Thesis Processing Office’s Template (NPS 
Req.) 

21 days 

Fri 10/3/14 

Fri 10/31/14 

Write Draft Chapters (NPS Req.) 

21 days 

Fri 10/3/14 

Fri 10/31/14 

Submit Draft Chapters to Thesis Processing (NPS 
Req.) 

1 day 

Fri 10/17/14 

Fri 10/17/14 
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Submit Draft Chapters to Advisor (NPS Req.) 

1 day 

Fri 10/31/14 

Fri 10/31/14 

Capstone Project Report Submitted for Chair 
Signature (NPS Req.) 

1 day 

Wed 11/5/14 

Wed 11/5/14 

Revise Final Based on Advisor Feedback (NPS 
Req.) 

3 days 

Wed 11/19/14 

Fri 11/21/14 

Obtain Advisor’s Approval of Final (NPS Req.) 

3 days 

Wed 11/26/14 

Fri 11/28/14 

Final Draft with Chair Revisions due to (TPO) 

1 day 

Fri 12/5/14 

Fri 12/5/14 

Final IPR 3 Rehearsal (advisors only) 

1 day 

Fri 12/5/14 

Fri 12/5/14 

Final IPR 3 (for stakeholders and faculty) 

1 day 

Fri 12/5/14 

Fri 12/5/14 

End of Final Quarter Stakeholder Approval 
(Control Gate) 

0 days Fri 12/12/14 Fri 12/12/14 


4.2 PROJECT REVIEWS 

The SEHV Capstone PM will monitor the project activities against the PMP to evaluate 
the actual performance and progress of the SEHV Capstone Project. In addition to the 
specific monitoring and control functions in section 2.3.1 above, the SEHV Capstone PM 
will coordinate, facilitate, and ensure stakeholder/team participation in quarterly IPR 
meetings. Through the IPRs and other controls described in this document, the SEHV 
Capstone PM will identify and document deviations from the PMP. The SEHV Capstone 
PM will collect and analyze the issues and determine the actions necessary to address the 
issues. Finally, the SEHV Capstone PM will take the requisite corrective action on any 
noncompliance issues and manage each to closure. 


4.3 DELIVERABLES 

The team will deliver both a SEHV Capstone final report and PMP. These artifacts serve 
as guides for the objectives and scope of the project. Three In-Progress Reviews (IPRs) 
will be conducted over the duration of this project. At each of the IPRs, the project team 
will review the project status and way ahead with the stakeholders. NPS faculty will also 
be present for the IPRs and ensure that quality SE is being performed for the project 
according to the NPS standards. The SEHV Capstone Project final report will capture the 
output of the planned activities from the capstone proposal and PMP. The final report is 
the major deliverable from this effort. Interim work products such as architectures and 
simulation results will be included in the report and will be available to stakeholders for 
follow on efforts. 
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Spring Quarter 

28 Mar Draft capstone proposal due to Advisor via Advisors’ Inbox in Sakai SI0810 
Resources. This is a team deliverable. 

4 Apr 1 st Meeting - meetings occur weekly after this date 

11 Apr Final capstone proposal due to the Advisors’ Inbox in Resources 
25 Apr Draft Project Plan due 

9 May Final Project Plan due 

6 Jun Chapter 1 and the outline of the Capstone Report are due (to the Advisors’ 

Inbox) 

6 Jun IPR 1 rehearsal (advisors only) 

13 Jun IPR 1 (all invited stakeholders and faculty) 

Summer Quarter 

25 Jul Chapters 2 and 3 of the Capstone Report are due (to the Advisors’ Inbox) 

5 Sep IPR 2 rehearsal (advisors only) 

12 Sep IPR 2 (all invited stakeholders and faculty) 

12 Sept Chapters 4 and 5 of the Capstone Report are due (to the Advisors’ Inbox) 
Fall Quarter 

17 Oct Initial draft of all sections of the Capstone Report is due (to the Advisors’ 
Inbox) 

1 Nov Initial draft of the Capstone Report is due to the Thesis Processing Office 
(TPO) for format check (via SharePoint (SP)) 

5 Nov Final draft of the Capstone Report (ready for advisor signature) is due to 
Heather Hahn (via SharePoint) for SE Department Chair Review 
5 Dec Final draft of the Capstone Report (incorporating any revisions from the 
Dept. Chair) is due to the TPO (facilitated by Heather Hahn) 

5 Dec Final IPR 3 rehearsal (for advisors) 

12 Dec Final IPR 3 (for stakeholders and faculty) 
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APPENDIX A ACRONYMS AND ABBREVIATIONS 


The following acronyms are used in this plan: 


Acronym 

Definition 

AHP 

Analytic Hierarchy Process 

ALI 

Applied Leading Indicators 

CCB 

Configuration Control Board 

CIMS 

Career Information Management System 

CM 

Configuration Management 

CMGR 

Configuration Manager 

COG 

Command Operating Guide 

CRM 

Customer Relationship Model 

DCO 

Defense Connect Online 

DC/ESE 

Data Center / Enterprise Services Environment 

DIMHRS 

Defense Information Management Human Resource 
System 

DOD 

Department of Defense 

DODAF 

Department of Defense Architecture Framework 

ERD 

Entity-Relationship Diagram 

FFBD 

Functional Flow Block Diagram 

IEEE 

Institute of Electrical and Electronics Engineers 

INCOSE 

International Council on Systems Engineering 

IPR 

In-Progress Review 

ISO 

International Organization for Standardization 

JF 

Joint Framework 

LSE 

Lead System Engineer 

MGMT 

Management 

MS 

Microsoft 

MSES 

Master of Science in Engineering Systems 

MSSE 

Master of Science in Systems Engineering 


168 




MYI 

Master Veteran Index 

NAVAIR 

Naval Air 

NPS 

Naval Postgraduate School 

NSIPS 

Navy Standard Integrated Performance System 

OY 

Operational View 

PM 

Project Manager 

PMP 

Project Management Plan 

POA&M 

Plan of Actions and Milestones 

PSE 

Program Support Engineer 

PSM 

Practical Software and Systems Measurement 

QA 

Quality Assurance 

QM 

Quality Management 

REQM 

Requirements Management 

ROM 

Rough Order of Magnitude 

SE 

Systems Engineering 

SEHV 

System Engineering Health and Visualization 

SELI 

System Engineering Leading Indicator 

SEMP 

System Engineering Management Plan 

SME 

Subject Matter Expert 

SP 

Share Point 

SPAWAR 

Space and Naval Warfare 

SSC-A 

Space and Naval Warfare Systems Center Atlantic 

TFFMPS 

Total Force and Fleet Manpower System 

TPO 

Thesis Processing Office 

VA 

Veteran Affairs 

V&V 

Verification and Validation 

WBS 

Work Breakdown Structure 

XML 

Extendible Markup Language 
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APPENDIX B SEHV CAPSTONE PROJECT TOOLS AND PLANS 


Table 6 provides a listing of the tools and plans that will be used to support the SEHV 
Capstone Project Management, Architecture and Simulation Modeling processes. 


Table 6 SEHV Capstone Project Management Tools and Plans 


Function 

Tool(s) Used 

Requirements Management 

Innoslate 

Schedule Management 

MS Project 

Communications Management 

Email, MS Excel 

Risk Management 

Risk Matrix, List 

Configuration Management 

Google Docs 

Document Repository 

Google Docs, Sakai Site, 
MS Outlook 

Architecture / Simulation 
Modeling 

Innoslate, MS Visio 
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APPENDIX C GLOSSARY 


- Configuration Control Board (CCB) 

The CCB is a group that assists with an organization’s overall network strategy. The 
CCB is a group of project stakeholders responsible for evaluating and approving or 
disapproving proposed changes to a system or strategy. The CCB is also responsible for 
prioritizing and scheduling approved change for upcoming releases of the system. 


- Entity-Relationship Diagram (ERD) 

An ERD is a data modeling technique that creates a graphical representation of the 
entities, attributes, and relationships within an information system. 


- Institute of Electrical and Electronics Engineers (IEEE) 

A large technical professional society promoting the development and application of 
electro-technology and allied sciences for the benefit of humanity, the advancement of 
the profession, and the well-being of our members. The IEEE fosters the development of 
standards that often become national and international standards. The organization 
publishes a number of journals, has many local chapters, and several large societies in 
special areas. 

- International Council on Systems Engineering (INCOSE) 

INCOSE is a not-for-profit membership organization founded in 1990. Their mission is 
to share, promote and advance the best of systems engineering from across the globe for 
the benefit of humanity and the planet. 


- International Organization for Standardization (ISO) 

The ISO is the largest developer of voluntary international standards. 


- Work Breakdown Structure (WBS) 

The WBS is a decomposition of all the work necessary to complete a project. 
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APPENDIX C. SSC-A’S CAO CONOPS AND IPT HANDBOOK 
STAKEHOLDER CHARACTERIZATION WRT SELIS 


Command Senior Leadership Staff 

Weekly Staff Meeting metrics monitor Command-wide activities that may impact 
IPT performance and vary based on Command priorities. Metrics include cycle 
times for key processes that require leadership attention to drive end-to-end 
efficiencies (CONOPS Pg. 38 SELI12) 

The Commanding Officer (CO) and Executive Director (ED), along with the 
governance entities shown in Figure 3, SSC-A Senior Leadership and Governance 
Structure, ensures the efficient flow of processes across the local competencies 
and the sharing of best processes with SPAWAR Systems Command 
(SPAWARSYSCOM) (CONOPS Pg. 4 SELI 12) 

CODE 01B DIRECTOR OF MANAGEMENT OPERATIONS 

The ED serves on the SPAWAR Business Board, institutes processes required to 
achieve programs and mission goals, and provides conflict resolution and balance 
between competency priorities and business unit priorities. (CONOPS Pg. 4 SELI 
12 ) 

The CO and ED, along with the governance entities shown in Figure 3, SSC-A 
Senior Leadership and Governance Structure, ensures the efficient flow of 
processes across the local competencies and the sharing of best processes with 
SPAWARSYSCOM (CONOPS Pg. 4 SELI 12) 

The director’s goal is to manage business operations around cross-functional end- 
to-end business processes that produce financial and other performance 
information with greater accuracy, reliability, and accessibility (CONOPS Pg. 4 
SELI 12) 

The ED focuses on ensuring cost-effective operations, maintains a supportive 
work environment capable of accomplishing all assigned roles and 
responsibilities, and oversees the technical performance of work performed within 
SSC-A. (CONOPS Pg. 4 SELI 9, 13, 16, 18) 

CODE 01E DIRECTOR OF INTEGRATION AND EFFICIENCY 
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The Director of Integration and Efficiency focuses on developing and deploying 
effective processes that operate across the command to achieve organizational 
efficiency. (CONOPS Pg. A-3 SELI12) 

CODE Oil INSPECTOR GENERAL 

Through the assessment of internal controls and related risks, the Office of 
Inspector General (OIG) prevents and detects fraud, waste, abuse, and 
mismanagement within the organization. (CONOPS A-3 SELI 9, 10) 

It provides services to management by objectively assessing activities, reporting 
on findings, providing independent advice and technical guidance, and 
recommending policies and processes changes to prevent and detect fraud, waste, 
abuse, and mismanagement. (CONOPS A-4 SELI 12) 

Business Portfolio 

The Business Portfolio Board (BPB) is the first of two major governance boards 
chartered by the ESC to focus on long-term visionary and strategic planning, 
processes, and resources for business execution. (CONOPS Pg. 5 SELI 12, 14) 

A Sub Portfolio’s (SP) responsibilities are to coordinate efforts across a collection 
of IPTs and other subportfolios to ensure maximum cost, schedule, and 
performance efficiencies. (IPT Handbook Pg. 2 SELI 9, 13, 16, 18) 

Coordinate projections of the resources that will be needed to meet anticipated 
requirements across fiscal years are collected in IPT Workbooks by the portfolios. 
The workbook provides personnel (series/level/geographic preference, KSA, 
special certifications), laboratories facilities, and contract requirements. 
(CONOPS Pg. 21 SELI 14) 

The portfolio structure enables the command to manage and have sound 
accountability of cost, schedule, and performance; ensure effective and timely 
delivery of products and services; and speak with a single command-voice in each 
business area. (CONOPS Pg. 6 SELI 9, 16, 18) 

Working with the Support Agreements Manager (SAM), functional managers 
review customer requests to determine SSC-A’s capability to provide the 
requested support, develop draft support agreements, determine the impact of the 
mission, and identify costs and resources to provide the support. (CONOPS Pg. 
18 SELI 9, 16) 
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At SSC-A, BPs are responsible for the overall execution of the tasking and 
funding (cost, schedule, and performance) in support of the customer, and for 
management and organization of the customer relationship.(IPT Handbook Pg. 1 
SELI9, 16, 18) 

BPs will coordinate across SSC-A and the SPAWAR Enterprise, as necessary, to 
ensure that the customer is provided the right work and right team, in the right 
place, and at the right cost and capability. (IPT Handbook Pg. 3 SELI 9, 16) 

BUSINESS PORTFOLIO MANAGER (BPM) 

The objectives of BP management are to optimize the mix of resources used to 
produce and deliver the end item and to schedule activities that achieve SSC-A’s 
goals within constraints imposed by customers, strategic objectives, and external 
factors. (CONOPS Pg. 8 SELI 11, 14, 18) 

BPMs and SPLs monitor the environment, analyze key factors, forecast resource 
needs, and develop business plans in order to prepare for and sustain future work. 
(CONOPS Pg. 8 SELI 11, 14) 

The BPMs shape the portfolio content and oversee the cost, schedule, and 
performance outcomes of their assigned portfolio. (CONOPS Pg. B-2 SELI 9, 16, 
18) 

BPMs ensure that cost-effective solutions are provided to customers with similar 
requirements. (IPT Handbook Pg. 2 SELI 9, 16) 

DEPUTY PORTFOLIO MANAGER (DPM) 

DPMs are responsible for daily BP operations, cost, schedule, and performance 
oversight; support the BP strategic efforts; and lead and advise the BP senior staff 
regarding the actions of the boards. (CONOPS Pg. B-2 SELI 9, 16, 18) 

PORTFOLIO SYSTEMS ENGINEER (PSE) 

Each PSE supports his/her aligned BPM during customer engagements to 
understand and then communicate the scope of the initial engineering 
requirements to the SPL and IPT Lead. (CONOPS Pg. B-2 SELI 1, 4, 5) 
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All products offered by their aligned BP adhere to Federal standards for 
interoperability, comply with Federal mandated architectures, and use system-of- 
systems engineering practices to advance the enterprise IT capability of the 
Warfighter. (CONOPS Pg. B-2 SELI 12, 17) 

SUBPORTFOLIO LEAD (SPL) 

Sub-IPT Leads may come from any competency but must possess the requisite 
KSAs and A&Es to effectively manage the cost, schedule, performance, and risks 
of their assigned project or tasks. (IPT Handbook Pg. B-5 SELI 9, 10, 16, 18) 

SPLs will assist with negotiated resource allocations (workforce Demand Signals, 
facilities, lab space, contracts, etc.) with the competencies. (IPT Handbook Pg. D- 
1 SELI 11, 14) 

The SPL is accountable for the cost, schedule, performance, and risk of the SP. 
(IPT Handbook Pg. 5 SELI 9, 13, 16) 

BPMs and SPLs monitor the environment, analyze key factors, forecast resource 
needs, and develop business plans in order to prepare for and sustain future work. 
(CONOPS Pg. 8 SELI 11, 14) 

BPMs and SPLs assess and forecast resource requirements for the competencies 
and BPs. (IPT Handbook Pg. 6 SELI 11,14) 

The Handshake is command-required documentation that defines the scope, 
schedule, cost, and performance of a project and is signed by both the designated 
Portfolio/SPL approver and the project sponsor or customer. (IPT Handbook Pg. 
18 SELI 9, 15, 16, 18) 

An SPL is responsible for one or more projects within a portfolio and for one or 
more IPTs. They are responsible for project cost, schedule, and performance 
oversight of the IPTs within the sub-portfolio. (CONOPS Pg. B-3 SELI 16) 

INTEGRATED PRODUCT TEAM (IPT) LEAD 

The IPT Lead works with IPT members to iteratively refine customer 
requirements, cost estimates, resource projections, and update Project Planning 
Monitoring and Control (P2MC) and Navy ERP. The IPT Lead collaborates with 


178 



support competencies during the Planning and Customer Requirements Phase. 
(CONOPS Pg. 31 SELI16) 

The Leads engage the appropriate subject matter experts (SME) to initiate the 
execution of the necessary process or processes as needed throughout the Project 
Life Cycle (PLC). (CONOPS Pg. 30 SELI 12) 

Risk Management (CONOPS Pg. 30 SELI 9, 10) 

The IPT Lead should have experience with the technology involved in the project, 
adequate familiarity in the AOR, and appreciation of IPT members’ capabilities. 
Detailed IPT Lead roles and responsibilities are defined in the IPT Handbook. 
(CONOPS Pg. B-3 SELI 8) 

The IPT Lead engages in early project management high level planning activities, 
and reviewing the description of the work contained in the Mission Alignment 
record. High level planning in project initiation typically includes high level 
refinement and management of requirements, risks, cost, and communications, 
and the development of a Work Breakdown Structure (WBS), Organizational 
Breakdown Structure (OBS), schedule, demand signals, charter, and Team 
Assignment Agreements (TAAs) (IPT Handbook Pg. 12 SELI 9, 10, 18) 

The IPT Lead tracks and monitors all project activities and controls deviations 
from the base-lined PMP. This is achieved by monitoring and controlling project 
execution, cost, schedule, scope, and risks for each activity in the project. (IPT 
Handbook Pg. 21 SELI 9, 10, 18) 

IPT Leads may come from any competency but must possess the requisite KSAs 
and A&Es to effectively manage the cost, schedule, performance, and risks of 
their assigned project or tasks. An IPT Lead must also follow the 6.0 Program and 
Project Management CDM. (IPT Handbook Pg. 12 SELI 9, 10, 18) 

The IPT Lead and Competency Managers will identify qualified team members 
(also known as a “Competency Member of an IPT” or CMol) through negotiated 
interaction and discussion of the required tasks (Refer to Section 3.1 for early 
communications of personnel resource requirements) (IPT Handbook Pg.17 SELI 
11 ) 
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The IPT Lead is responsible for assisting and negotiating with competency 
representatives for personnel resource requirements or Demand Signal for the IPT 
(IPT Handbook Pg.17 SELI11) 

The IPT Lead will also provide feedback to Competency Managers for employee 
accomplishments for performance evaluations and for any schedule or resource 
deviations which have the potential to affect the competency. (IPT Handbook Pg. 
22 SELI 11, 18) 

The IPT Lead manages complex change by providing vision, skills, demand 
signal, resources, and action plans to effectively respond to changing 
requirements. (IPT Handbook Pg. C-2 SELI 11) 

The IPT Lead has the authority to size the IPT in support of the requirements. 

The IPT Lead is responsible for coordinating with the portfolios, SPs, and 
competencies to resolve issues due to scope, funding, performance, and resource 
priorities. (IPT Handbook Pg. D-2 SELI 11) 

The IPT Lead assesses, forecast, and maintain resource demand signals for a 2- 
year outlook (execution +1 year). (IPT Handbook Pg. 7 SELI 11) 

The IPT Lead communicates resource status and ensure efficient resource 
utilization (IPT Handbook Pg. 7 SELI 11) 

The IPT Lead engages all competencies necessary to develop a project resources 
plan; this plan defines resource requirements including personnel, facilities 
(administrative, warehouse, laboratory), IT, and materials. (IPT Handbook Pg. 13 
SELI 11) 

Each IPT Lead collaborates early and often with Competency Managers to 
negotiate IPT membership and obtain personnel with the required Knowledge, 
Skills, and Abilities/Assignment & Experience (KSAs/A&E) from 
organizationally-aligned competency resources. (IPT Handbook Pg.16 SELI 11) 

The IPT Lead is responsible for assisting and negotiating with competency 
representatives for personnel resource requirements or Demand Signal for the 
IPT. (IPT Handbook Pg. 16 SELI 11) 
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IPT members or CMol shall perform their work in accordance with cost, 
schedule, performance, and risk parameters set by the IPT Lead utilizing common 
work processes and the operational policies, procedures, and standards of the 
competency. (IPT Handbook Pg. 22 SELI9, 12, 18) 

The IPT Lead engage all members in the IPT process by soliciting inputs and 
applying active listening skills (IPT Handbook Pg. 24 SELI 12) 

The IPT Lead identify, document, mature, and use standardized processes across 
projects to enable quality, speed, agility, and value (IPT Handbook Pg. C-3 SELI 
12 ) 

The IPT Lead develops a detailed Project Management Plan (PMP) to define how 
the project is to be executed, monitored and controlled, and closed. PMP planning 
documentation includes the agreed-to customer requirements; project scope, 
schedule, and resource requirements; project WBS organizational breakdown 
structure (OBS); plans for project data and reporting; plans for communications; 
other documents necessary to monitor the project cost, schedule, and 
performance; control variances from the project plan; produce and deliver the end 
product; and closeout the project. (CONOPS Pg. 29 SELI 9, 13, 16, 18) 

IPT Leads and Competency Leads confer frequently to evaluate performance of 
competency members and address performance issues to ensure that CMol 
performance is in accordance with the TAAs, meets the IPT requirements, 
respective competency policies and processes, and that competency member 
development needs are being addressed.(CONOPS Pg. 30 SELI 13) 

Manage day-to-day cost, schedule, performance, and risk elements of the IPT. 
(IPT Handbook Pg.7 SELI 9, 13, 16, 18) 

Collect and measure performance data (Health Metrics) and report results to the 
SPL and BPM overseeing the project (IPT Handbook Pg. 7 SELI 13) 

Focus on needs of customer: Identifying Customer Needs - Measuring 
Performance - Establishing Project Controls (IPT Handbook Pg. C-l SELI 13) 

The IPT Lead gathers and measure performance data (IPT Handbook Pg. C-2 
SELI 13) 
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The IPT Lead report performance measures to portfolio and competency 
leadership. (IPT Handbook Pg. C-2 SELI13) 

Measure performance, track performance measurements, and use CMMI 
processes (IPT Handbook Pg. C-4 SELI 13) 

IPT Lead selection occurs early in the project. This allows the IPT Lead to 
participate in key startup decisions such as project resource planning. The IPT 
Lead is the hands-on, day-to-day manager of the project and the person most 
directly responsible for the success of the project. (CONOPS Pg. 26 SELI 14) 

The IPT Lead engages the competencies with the preponderance of the work to 
develop a project resources plan; this plan defines resource requirements 
including personnel, facilities, warehousing and laboratory space. (CONOPS Pg. 
26 SELI 14) 

IPT Lead sends the ROM cost to the customer and discusses the resourcing 
approach to ensure funds are provided as needed to support the project. 
(CONOPS Pg. 27 SELI 14) 

The IPT Lead develops a detailed Project Management Plan (PMP) to define how 
the project is to be executed, monitored and controlled, and closed. PMP planning 
documentation includes the agreed-to customer requirements; project scope, 
schedule, and resource requirements; project WBS organizational breakdown 
structure (OBS); plans for project data and reporting; plans for communications; 
other documents necessary to monitor the project cost, schedule, and 
performance; control variances from the project plan; produce and deliver the end 
product; and closeout the project. (CONOPS Pg. 29 SELI 14) 

Communicate resource status and ensure efficient resource utilization (IPT 
Handbook Pg. 7 SELI 14) 

The IPT Lead engages all competencies necessary to develop a project resources 
plan; this plan defines resource requirements including personnel, facilities 
(administrative, warehouse, laboratory), IT, and materials. (IPT Handbook Pg. 13 
SELI 11, 14) 

IPT Leads will gather requirements and provide to the Competency Lead in 
support of facilities requests. The Competency Lead will submit request(s) for 
facilities (lab, warehouse, and administrative) resources in support of the IPT. 
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(SPAWARSYSCENLANT 5900/2 (Rev. 06/12)) (IPT Handbook Pg. D-2 SELI 
14) 

The IPT Lead conducts lessons learned workshops/activities and post project 
reviews to ensure data collected is added to documents such as the PMP for 
complete and accurate documentation. (CONOPS Pg. 37 SELI 15) 

The PMP will be developed by the IPT Lead with input from the customer and 
other key stakeholders. (CONOPS Pg. 19 SELI 15) 

The IPT Lead tracks and monitors all project activities and controls deviations 
from the baseline PMP. This is achieved by monitoring and controlling project 
execution, cost, schedule, scope, and risks for each activity in the project. 
(CONOPS Pg. 29 SELI 9, 16, 18) 

The IPT Lead conducts lessons learned workshops/activities and post project 
reviews to ensure data collected is added to documents such as the PMP for 
complete and accurate documentation. This will allow for full accounting of 
information and can expedite and cut costs of future projects. (CONOPS Pg. 37 
SELI 9, 16) 

The IPT Lead engages in early project management high level planning activities, 
and reviewing the description of the work contained in the Mission Alignment 
record. High level planning in project initiation typically includes high level 
refinement and management of requirements, risks, cost, and communications, 
and the development of a Work Breakdown Structure (WBS), Organizational 
Breakdown Structure (OBS), schedule, demand signals, charter, and Team 
Assignment Agreements (TAAs). (IPT Handbook Pg. 12 SELI 9, 16, 18) 

Each IPT shall have a designated leader who is accountable for the cost, schedule, 
performance, and risk of the IPT (IPT Handbook Pg. 6 SELI 9, 16, 18) 

A preliminary schedule and ROM cost estimate (can be used to for Handshake 
Artifact estimates) are established; schedules and cost estimates become more 
accurate as more information becomes available. (IPT Handbook Pg. 13 SELI 9, 
16, 18) 

The IPT Lead tracks and monitors all project activities and controls deviations 
from the base-lined PMP. This is achieved by monitoring and controlling project 
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execution, cost, schedule, scope, and risks for each activity in the project. (IPT 
Handbook Pg. 21 SELI9, 16, 18) 

The IPT Lead will report up his or her IPT chain of command business 
accomplishments based on cost, schedule, performance, and risk, and will be held 
accountable for those elements (IPT Handbook Pg. 22 SELI 9, 16, 18) 

IPT Leads may come from any competency but must possess the requisite KSAs 
and A&Es to effectively manage the cost, schedule, performance, and risks of 
their assigned project or tasks. (IPT Handbook Pg. B-3 SELI 9, 16, 18) 

IPT TECHNICAL LEAD 

This role ensures that proper technical activities are being conducted including 
determining and recommending the correct technical competency staffing and 
resources necessary for work execution to IPT Lead, overseeing technical work 
execution, reviewing deliverables for technical content and correctness, providing 
signatures on technical documentation where necessary, and ensuring proper 
technical reviews are being conducted in accordance with command directives. 
(CONOPS Pg. B-4 SELI 6) 

The IPT Technical Lead provides technical guidance as necessary in accordance 
with command initiatives and directives, advises the IPT on technical risks, and 
issues and makes recommendations as a direct support to the IPT Lead. 
(CONOPS Pg. B-4 SELI 10) 

The Leads engage the appropriate subject matter experts (SME) to initiate the 
execution of the necessary process or processes as needed throughout the PLC. 
(CONOPS Pg. 30 SELI 12) 

Project-specific process development and management is the responsibility of the 
IPT Lead. (CONOPS Pg. D-3 SELI 12) 

This role ensures that proper technical activities are being conducted including 
determining and recommending the correct technical competency staffing and 
resources necessary for work execution to IPT Lead, overseeing technical work 
execution, reviewing deliverables for technical content and correctness, providing 
signatures on technical documentation where necessary, and ensuring proper 
technical reviews are being conducted in accordance with command directives. 
(CONOPS Pg. B-4 SELI 14, 15) 
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5.0 Competency 


Competencies provide personnel resources for project execution. However, 
competencies often find it necessary to contract with industry partners who 
provide contractor employees to augment competency resources. A Govemment- 
to-industry ratio is not prescribed on projects; however how effectively each 
project uses competency resources is monitored. Sourcing and personnel 
decisions must be carefully examined to ensure there is adequate and qualified in- 
house Government expertise involved in and providing oversight on every 
project. (CONOPS Pg. G-3 SELI11) 

Command’s competencies are responsible for providing personnel resources, 
either government or industry, required by an IPT to execute the BP’s work 
efforts. (IPT Handbook Pg. 2 SELI 11) 

Communicate resource requirements (Demand Signal). (IPT Handbook Pg. 6 
SELI 11) 

The IPT Lead engages all competencies necessary to develop a project resources 
plan; this plan defines resource requirements including personnel, facilities 
(administrative, warehouse, laboratory), IT, and materials. (IPT Handbook Pg. 13 
SELI 11) 

It should be noted that SSC-A competencies provide standardized command 
processes, templates, tools, and other process assets for IPTs’ use. (IPT Handbook 
Pg. 20 SELI 12) 

Technical authorities provide technical expertise in the warranted competency 
area as well as establish standards, requirements, and processes to ensure 
consistency and product conformity. (CONOPS Pg. 10 SELI 12) 

The LCL advises the NCL of competency demand, workforce development needs, 
work processes, and operating standards appropriate to the elements of the 
competency to which they are assigned (CONOPS Pg. D-2 SELI 12) 

The Competency Board (CB) is the second of the two major governance boards 
chartered by the ESC to focus on long-term visionary and strategic planning, 
processes, and resources for competency execution. This board governs the 
competencies and provides strategic and tactical direction, guidance, oversight, 
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and conflict resolution to ensure consistent handling of all resources across the 
Command. (CONOPS Pg. 5 SELI14) 

Competencies provide the resources—people, processes, facilities, contracts, and 
tools—needed by the IPTs to deliver products and services to customers. 
(CONOPS Pg. 8 SELI 14) 

The IPT Lead engages the competencies with the preponderance of the work to 
develop a project resources plan; this plan defines resource requirements 
including personnel, facilities, warehousing and laboratory space. (CONOPS Pg. 
26 SELI 14, 16) 

At SSC-A, the competencies are the organizational elements that provide 
resources (people, processes, and tools) to carry-out the Command’s work. (IPT 
Handbook Pg. 1 SELI 14) 

Under the IPT process, the Engineering Competency provides systems 
engineering and core engineering skills to other SSC-A competencies. Core skills 
are provided in support of warfighting technologies and systems; system designs; 
systems engineering within and across platforms, domains, and missions; test, 
evaluation, and certification of systems and capabilities; analyzing system data; 
determining and implementing corrective actions; ensuring security, safety, 
affordability, reliability, maintainability, usability, and availability of systems; 
and performing engineering assessments, cost estimations, and investigations. 
(CONOPS Pg. E-4 SELI 16) 

Obtain cost estimates from appropriate competencies (IPT Handbook Pg. 8 SELI 
16) 

TIER 1 LOCAL COMPETENCY LEAD (LCL) 

The Tier 1 Local Competency Leads (LCL) establishes Tier 2 Competency Lead 
(CL), Tier 3 Competency Manager (CM), and Tier 4 Competency Supervisor 
(CS) positions to support competency execution requirements. 

LCLs consult with the SPAWAR National Competency Leads (NCL), who have 
overall authority and responsibility for all SPAWAR competencies, to ensure 
consistent competency KSAs, processes, and tools are developed in accordance 
with NCL guidance and are available as needed across SSC-A (CONOPS Pg. D-2 
SELI 12) 
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LEAD SYSTEMS ENGINEER (LSE) 

Performing engineering reviews to ensure proper interpretation and 
implementation of engineering processes and procedures used to produce end 
item products. (CONOPS Pg. D-3 SELI7, 12) 

The LSE ensures, through disciplined systems engineering, that all products and 
services adhere to competency standards, comply with federally mandated 
architectures, and advance the enterprise information technology capability of the 
Warfighter. (CONOPS Pg. D-2 SELI 17) 

CONTINUOUS PROCESS IMPROVEMENT (CPI) LEAD 

The CPI Lead collaborates within and across the SSC-A leadership team 
regarding improvement requirements, planning, and execution. The CPI lead 
coordinates and manages Lean Six Sigma black and green belt assignments as 
well as training requirements to implement improvements. 

The CPI Executive Steering Group (ESG) represents the interests of portfolios, 
competencies, and command-level process leads to ensure that a structured 
method is used to consider proposed changes to command processes. (CONOPS 
Pg. 21 SELI 12) 

PROCESS OWNER 

The Process Owner can be assigned at any level of a competency. The Process 
Owner is responsible for overseeing and managing compliance with the 
competency’s processes, procedures, data models, policies, metrics, and 
technologies. (CONOPS Pg. D-3 SELI 12) 

The Process Owner ensures the process is used as required and the control plan is 
implemented and monitored. (CONOPS Pg. D-3 SELI 12) 

PROCESS MANAGER 

The Process Manager plans and coordinates activities required to perform, 
monitor and report on the process. (CONOPS Pg. D-3 SELI 12) 

The Process Manager collects process measurement data to ensure the expected 
output is achieved. (CONOPS Pg. D-3 SELI 13) 
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COMPETENCY ADMINISTRATIVE SUPPORT 


Support Services Specialist (SSS) functions are similar to the AO functions with 
specific focus on the assigned competency’s requirements. (CONOPS Pg. D-4 
SELI 1,4,5) 

TIER 2 COMPETENCY LEAD (CL) 

The Tier 1 Local Competency Leads (LCL) establish Tier 2 Competency Lead 
(CL), Tier 3 Competency Manager (CM), and Tier 4 Competency Supervisor 
(CS) positions to support competency execution requirements. (CONOPS Pg. D-4 
SELI 11) 

The Competency Lead will submit request(s) for facilities (lab, warehouse, and 
administrative) resources in support of the IPT. (SPAWARSYSCENLANT 5900/ 
2 (Rev. 06/12)) (IPT Handbook Pg. D-2 SELI 11) 

IPT Leads and Competency Leads confer frequently to evaluate performance of 
competency members and address performance issues to ensure that CMol 
performance is in accordance with the TAAs, meets the IPT requirements, 
respective competency policies and processes, and that competency member 
development needs are being addressed. (CONOPS Pg. 30 SELI 13) 

The CL is responsible for all of the competency’s resources: personnel including 
their development and assignment, processes, and tools. (CONOPS Pg. D-4 SELI 
14) 

TIER 3 COMPETENCY MANAGER (CM) 

The Tier 1 Local Competency Leads (LCL) establishes Tier 2 Competency Lead 
(CL), Tier 3 Competency Manager (CM), and Tier 4 Competency Supervisor 
(CS) positions to support competency execution requirements. (CONOPS Pg. D-4 
SELI 11) 

The IPT Lead and Competency Managers will identify qualified team members 
(also known as a “Competency Member of an IPT” or CMol) through negotiated 
interaction and discussion of the required tasks (Refer to Section 3.1 for early 
communications of personnel resource requirements). (IPT Handbook Pg. 17 
SELI 11) 
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If current competency resources are not available to support the project, the 
Competency Manager will evaluate the need to develop or hire the required 
resources via the Demand Signal Process. (IPT Handbook Pg. 17 SELI11) 

The IPT Lead will also provide feedback to Competency Managers for employee 
accomplishments for performance evaluations and for any schedule or resource 
deviations which have the potential to affect the competency. (IPT Handbook Pg. 
22 SELI 11, 18) 

Each IPT Lead collaborates early and often with Competency Managers to 
negotiate IPT membership and obtain personnel with the required Knowledge, 
Skills, and Abilities/Assignment & Experience (KSAs/A&E) from 
organizationally-aligned competency resources. (IPT Handbook Pg.16 SELI 11) 

TIER 4 COMPETENCY SUPERVISOR (CS) 

The Tier 1 Local Competency Leads (LCL) establishes Tier 2 Competency Lead 
(CL), Tier 3 Competency Manager (CM), and Tier 4 Competency Supervisor 
(CS) positions to support competency execution requirements. (CONOPS Pg. D-4 
SELI 11) 

The Team Assignment Agreement (TAA) establishes interrelationships and 
responsibilities between the competency performing individual, the IPT 

Leadership, and the individual’s Competency Supervisor. (IPT Handbook Pg. 16 
SELI 11) 

FIRST-LINE SUPERVISOR 

The Team Assignment Agreement (TAA) establishes interrelationships and 
responsibilities between the competency performing individual, the IPT 

Leadership, and the individual’s Competency Supervisor. (IPT Handbook Pg. 16 
SELI 11) 

Lirst-Line Supervisor provides additional Performance Management information. 
(CONOPS Pg. 31 SELI 13) 

CONTRACTING OFFICER 'S REPRESENTATIVE (COR) 

The COR works in conjunction with the IPT lead to ensure the IPT is resourced 
efficiently to execute the COR function. (IPT Handbook Pg. 14 SELI 11, 14) 
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The COR Resource Guide (CRG) has been established to assist the COR and the 
IPT lead with assigning roles to CMol personnel in an effort to effectively and 
efficiently execute the COR functional role. (IPT Handbook Pg. 14 SELI11) 

To assist in monitoring contractor performance, the IPT requests, competency 
nominates, and the Contracts Competency appoints CORs for all services task 
orders. (CONOPS Pg. D-5 SELI 13) 

Additionally, the COR develops a Contract Performance Assessment Reporting 
System (CPARS) Draft Approval Document (CDAD) for all task orders and an 
actual CPARS evaluation for all orders over $1M. (CONOPS Pg. D-6 SELI 13) 

Particular project cost estimate should be tailored to meet the requirements of the 
level of complexity of the technical order (TO). (IPT Handbook Pg. 14 SELI 16) 


Integrated Product Team 

They perform under the direction of an IPT Lead to produce products and services 
for customers based on requirements contained in a statement of work or similar 
document. The purpose of the IPT Execution is to deliver end products that meet 
customer requirements on schedule and within the budget. Engineering members 
of the IPT use engineering design processes to decompose high-level 
requirements into functional requirements. The S&T and Engineering members of 
the IPT collaborate to develop portfolio awareness of new technologies and to 
match these technologies with IPT requirements. Logistics and Fleet Support 
members of the IPT provide project configuration management to ensure the end- 
product complies with the design developed in the System Requirements Design 
Phase. (CONOPS Pg. 11 SELI 1, 4, 5, 18) 

The Science and Technology (S&T) and Engineering members of the IPT 
collaborate to develop portfolio awareness of new technologies and to match 
these technologies with IPT requirements. (CONOPS Pg. 32 SELI 8) 

The S&T Competency supports the IPT with the transition of new technology 
concepts and components to the engineering community for incorporation into the 
end-product. (CONOPS Pg. 33 SELI 8) 

The sponsor’s process or product addresses or makes provisions for meeting SSC- 
A’s key process requirements, the IPT may be authorized to substitute the 


190 



sponsor’s process/product in lieu of the command’s standardized process/asset. 
(IPT Handbook Pg. 20 SELI12) 

IPT members or CMol shall perform their work in accordance with cost, 
schedule, performance, and risk parameters set by the IPT Lead utilizing common 
work processes and the operational policies, procedures, and standards of the 
competency. (IPT Handbook Pg. 22 SELI 12, 18) 

IPTs observe, measure, and report project performance in the P2MC and Navy 
ERP systems. Project monitoring above the SSC-A level is supported by 
Enterprise Cost Management Framework (ECMF) tagging. ECMF provides a 
common language about resources requirements, work, products, and services 
across the Navy. Tagging Navy ERP records enables visibility into financial 
resources at all levels of the Navy. (CONOPS Pg. 29 SELI 13) 

Engineering members of the IPT use engineering design processes to decompose 
high-level requirements into functional requirements, identify system design 
specifications, and prepare design level artifacts that represent the end-product to 
be delivered to the customer. Engineering activities include developing a logical 
solution and documenting the solution design; developing, applying, and 
documenting the physical design and platform requirements; designing and 
documenting the software and hardware components of the solution; and 
selecting, collecting, analyzing, and reporting performance data related to the 
products developed and processes implemented. (CONOPS Pg. 32 SELI 13) 

Execute task, project, and/or program responsibilities for cost, schedule, and 
performance (IPT Handbook Pg. 4 SELI 13, 18) 

Each IPT shall have a designated leader who is accountable for the cost, schedule, 
performance, and risk of the IPT. (IPT Handbook Pg. 6 SELI 13, 18) 

Department of Navy (DON) life cycle management organization methodology 
will employ IPTs that will manage and integrate critical processes ensuring that 
products optimize performance and long-term cost tradeoffs across all equipment, 
weapon, and information technology systems, and organizational sub-elements. 
(IPT Handbook Pg. B-l SELI 13) 

This determination involves defining the work request as one or more projects and 
assigning projects to IPTs; ensuring that project scope and resource requirements 
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are understood and documented; and engaging all involved parties in scheduling, 
estimating, and accepting the resulting project. (CONOPS Pg. 26 SELI14) 

With Corporate Operations the IPT manages System Administrators (SAs) and 
Memorandums of Understanding (MOUs) and plans for project-specific resources 
and administrative services. (CONOPS Pg. 26 SELI 14) 

Resources may include personnel, facilities, IT infrastructure, and other resources 
required to execute the IPT request. (CONOPS Pg. 11 SELI 11, 14) 

For those projects which have, or develop supplemental plans to address in detail 
any of the below areas (e.g., the project has or will develop a separate Risk 
Management Plan to identify, classify, categorize, and mitigate project risks), that 
section of the PMP will contain a reference to the more detailed plan (e.g., The 
project’s Risk Management Plan (document ID xxx) provides detailed 
information on the project’s risk identification and mitigation strategy). (IPT 
Handbook Pg. 19 SELI 15) 

The IPT leverages the ISEA service desk to provide support services, system 
administration, and provisioning that includes multi-tier support, troubleshooting, 
and training. The process includes analysis to determine the type and scope of 
effort required: hardware or software maintenance or documentation updates. 
(IPT Handbook Pg. 36 SELI 15) 

The collective IPT must review and finalize team documentation, submit 
documentation for approval to the next higher level IPT, and commit to the 
approved documentation. Documentation, which may be included in review and 
finalized by the team members, includes the IPT’s Charter, PMP, Risk 
Management Plan, Measurement and Analysis Plan, budget allocations, schedule, 
performance parameters, project milestones, and Quality Assurance Plan , among 
others pertinent to the IPT. (IPT Handbook Pg. 21 SELI 15) 

With Finance, the IPT performs business case analysis, models and refines 
original cost estimates, ensures business and financial compliance, and prepares 
funding documents. (CONOPS Pg. 27 SELI 16) 

Effectively monitoring and controlling requires capturing the earned value (EV) 
of work performed, actual costs (AC) of work performed, and planned value (PV) 
of work scheduled for the measuring period. (CONOPS Pg. 29 SELI 16, 18) 
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The IPT Lead works with IPT members to iteratively refine customer 
requirements, cost estimates, resource projections, and update P2MC and Navy 
ERP (CONOPS Pg. 31 SELI 16) 

Execute task, project, and/or program responsibilities for cost, schedule, and 
performance (IPT Handbook Pg.4 SELI 9, 16, 18) 

IPT members or CMol shall perform their work in accordance with cost, 
schedule, performance, and risk parameters set by the IPT Lead utilizing common 
work processes and the operational policies, procedures, and standards of the 
competency. (IPT Handbook Pg. 22 SELI 9, 16, 18) 

Project Manager 

The purpose of PM is to manage cost, schedule, and performance of product or 
service delivery based on an established set of requirements; establish and 
maintain a group of plans that define project activities execution; provide 
understanding of the project progress; and take appropriate corrective actions 
when project performance deviates significantly from the project plan. (CONOPS 
Pg. 29 SELI 9, 16, 18) 


Table 7: Stakeholder Analysis 


Active Stakeholder (User) 

Concerned SELI(s) 

COMMAND SENIOR 
LEADERSHIP STAFF 

#12 - Process Compliance Trends 

SSC-A EXECUTIVE OFFICER 

#12 - Process Compliance Trends 

CODE 01B DIRECTOR OF 
MANAGEMENT 

OPERATIONS 

#9 - Risk Exposure Trends (Cost & Schedule) 

#12 - Process Compliance Trend 
#13 - Technical Measurement Trends 
#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure Trends 

CODE 0IE DIRECTOR OF 
INTEGRA TION AND 
EFFICIENCY 

#12 - Process Compliance Trends 

CODE Oil INSPECTOR 
GENERAL 

#9 - Risk Exposure Trends (Cost & Schedule) 

#10 - Risk Treatment Trends 
#12 - Process Compliance Trend 

BUSINESS PORTFOLIO 

#9 - Risk Exposure Trends (Cost & Schedule) 
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H #12 - Process Compliance Trends 
#13 - Technical Measurement Trends 
#14 - Facility Equipment Availability 
■ #16 - System Affordability Trends 

#18 - Schedule and Cost Pressure Trends 

BUSINESS PORTFOLIO 
MANAGER (BPM) 

#9 - Risk Exposure Trends (Cost & Schedule) 

#11 - Systems Engineering Staffing and Skills 

Trends 

#14 - Facility Equipment Availability 
#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure Trends 

DEPUTY PORTFOLIO 
MANAGER (DPM) 

#9 - Risk Exposure Trends (Cost & Schedule) 

#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure Trends 

PORTFOLIO SYSTEMS 
ENGINEER (PSE) 

#1 - Requirements Trends 
#4 -Requirements Validation Trends 
#5 - Requirements Verification Trends 
#12 - Process Compliance Trends 
#17 - Architecture Trends 

SUBPORTFOLIO LEAD (SPL) 

#9 - Risk Exposure Trends (Cost and Schedule) 

#10 - Risk Treatment Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

#13 - Technical Measurement Trends 

#14 - Facility Equipment Availability 

#15 - Defect or Error Trend Specification (Artifact 

Documentation) 

#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure 

INTEGRATED PRODUCT 
TEAM (IPT) LEAD 

#1 - Requirements Trends 

#2 - System Definition Change Backlog Trends 

#3 - Interface Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

#9 - Risk Exposure Trends (Cost and Schedule) 

#10 - Risk Treatment Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

#12 - Process Compliance Trend 
#13 - Technical Measurement Trends 
#14 - Facility and Equipment Availability 
#15 - Defect or Error Trends (Artifact 

Documentation) 

#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure Trends 
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IPT TECHNICAL LEAD 

#6 - Work Product Approval Trends 

#9 - Risk Exposure Trends (Cost and Schedule) 

#10 - Risk Treatment Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

#14 - Facility and Equipment Availability Trends 
#15 - Defect or Error Trends Specification (Artifact 
Documentation) 

5.0 COMPETENCY 

#11 - Systems Engineering Staffing and Skills 

Trends 

#12 - Process Compliance Trend 
#14 - Facility Equipment Availability 
#16 - System Affordability Trends 

TIER 1 LOCAL 

COMPETENCY LEAD (LCL) 

#12 - Process Compliance Trends 

LEAD SYSTEMS ENGINEER 
(LSE) 

#7 - Review Action Closure Trends 
#12 - Process Compliance Trends 
#17 - Architecture Trends 

CONTINUOUS PROCESS 
IMPROVEMENT (CPI) LEAD 

#1 - Requirements Trends 

#2 - System Definition Change Backlog Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

#12 - Process Compliance Trend 

PROCESS OWNER 

#12 - Process Compliance Trends 
#13 - Technical Measurement Trends 

PROCESS MANAGER 

#12 - Process Compliance Trends 
#13 - Technical Measurement Trends 

COMPETENCY 
ADMINISTRATIVE SUPPORT 

#1 - Requirements Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

TIER 2 COMPETENCY LEAD 
(CL) 

#1 - Requirements Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

#13 - Technical Measurement Trends 
#14 - Facility and Equipment Availability 

TIER 3 COMPETENCY 
MANAGER (CM) 

#1 - Requirements Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

#18 - Schedule and Cost Pressure 

TIER 4 COMPETENCY 
SUPERVISOR (CS) 

#1 - Requirements Trends 

#4 - Requirements Validation Trends 


195 















#5 - Requirements Verification Trends 

#11 - Systems Engineering Staffing and Skills 

Trends 

FIRST-LINE SUPERVISOR 

#11 - Systems Engineering Staffing and Skills 

Trends 

#13 - Technical Measurement Trends 

CONTRACTING OFFICER'S 
REPRESENTATIVE (COR) 

#11 - Systems Engineering Staffing and Skills 

Trends 

#13 - Technical Measurement Trends 

#14 - Facility and Equipment Availability Trends 

#16 - System Affordability Trends 

Integrated Product Team 

#1 - Requirements Trends 

#2 - System Definition Change Backlog Trends 

#4 - Requirements Validation Trends 

#5 - Requirements Verification Trends 

#8 - Technology Maturity Trends 

#9 - Risk Exposure Trends (Cost and Schedule) 

#11 - Systems Engineering Staffing and Skills 

Trends 

#12 - Process Compliance Trend 

#13 - Technical Measurement Trends 

#14 - Facility and Equipment Availability Trends 

#15 - Defect or Error Trend Specification (Artifact 

Documentation) 

#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure Trends 

Project Manager 

#9 - Risk Exposure Trends (Cost and Schedule) 

#16 - System Affordability Trends 
#18 - Schedule and Cost Pressure 
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APPENDIX D. RESEARCH QUESTIONS AND ANSWERS 


Working with the stakeholders’ requirements and current state of technology, the 
SEHY team developed system engineering research questions that guided the research 
and the system conceptual design. These questions formed the basis for understanding 
stakeholders’ needs and system goal of this project. 

1. Data Sources 

• Which tools and supporting processes capture the data elements 
needed to ingest into the SEHV solution? 

Answer. Tools such as NERP, JIRA, IBM Rational, TAA tool, 
Demand Signal Tool, TWMS, Risk Exchange, and the Defense 
Civilian Personnel Data System (DCPDS) would support Risk 
Exposure Trends, SE Staffing and Skills Trends, Process 
Compliance Trends, Technical Measurement Trends, and System 
Affordability Trends. 

• What engineering tools are being leveraged? 

Answer. NERP, Risk Exchange, DCPDS, IBM Rational, JIRA, 
TAA, Demand Signal Tool, and TWMS would be leveraged to 
support SELI trending. SELIs not expanded on within this capstone 
report would need additional SE / SE management tools or 
applications to be leveraged. 

• What data sources are available? 

Answer. Data elements gathered from tools such as NERP, JIRA, 
IBM Rational, TAA tool, Demand Signal Tool, TWMS, Risk 
Exchange, and DCPDS would support Risk Exposure Trends, SE 
Staffing and Skills Trends, Process Compliance Trends, Technical 
Measurement Trends, and System Affordability Trends. 

• What data elements are needed? 

Answer. Data elements are listed within Chapter IV under each 
selected SELI. 

• What tool data is most dependable or valuable? 

Answer. Recommended data elements listed in Chapter IV under 
each SELI would be the most valuable to display trending data 
over time. 
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2 . 


Data Collection 


• How is data currently being captured? 

Answer. Data is currently being captured in SE / SE management 
Tools that SSC-A is using; however, this data is not being gathered 
in a consolidated location to produce trends of SELL Limited 
manual data call processes take place currently to capture data to 
determine project health and status. 

• How will the automatic collection work? 

Answer. Automatic collection would have to be done by 
middleware software that collects necessary data elements from 
the SE / SE management tools and stored in a database where 
historical and current SELI trends would be able to be produced. 
Automatic data collection is part of the SEHV methodology which 
includes capture, store, analyze, and display. Physical architecture 
specifies how data will be captured in future work. 

• If the data cannot be collected automatically, can the SELI data 
elements be collected on some sort of schedule? 

Answer. Chapter V recommends that SE / SE management would 
have to utilize mandated tools at least once a month to have up to 
date information. Data collection should be performed once a 
month to have current information. An automated process is 
preferred to reduce recurring loops and human error. 

• How easily can one collect any specific piece of data? 

Answer. Future work would have to include analyzing SE / SE 
management tools more in depth to see how easily it would be to 
collect specific pieces of data. Transforming the format of this data 
may need to take place in order to be compatible with the SEHV 
system. 


3, Data Storage 

• How are these data elements (individual pieces of data) related to one 
another? 

Answer. Data elements for SELI trends were found to be related to 
each other in different ways depending on the specific data 
elements in question. Data elements from NERP can display the 
amount of man-hours used for a particular SE activity which 
would support SE Staffing and Skills trends as well as System 
Affordability trends. Data elements tied to Technical Measurement 
trends would also fuel Risk Exposure trends. Overall these data 
elements for SELIs are related by producing visual trends that 
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would display the overall health of a project with respect to SE / 
SE management activities. 

What questions will be asked of the data? 

Answer. Per the SELI Guide version 2.0, data elements for Risk 
Exposure trends would pose questions such as: 

• Is the risk exposure going to impact the system solution? 

• Is the SE effort managing the exposure successfully? 

Per the SELI Guide version 2.0, data elements for SE Staffing and 
Skills trends would pose questions such as: 

• Is SE effort being applied to the project activities consistent 
with proven organizational or industry practice? 

• Do the staff members have the appropriate skills and 
experience to achieve assigned tasks? 

• Is the personnel turnover a reason for concern? 

Per the SELI Guide version 2.0, data elements for Process 
Compliance trends would pose questions such as: 

• To what extent are the SE processes in place and being used on 
the project? 

Per the SELI Guide version 2.0, data elements for Technical 
Measurement trends would pose questions such as: 

• To what extent are the performance parameters feasible and 
being achieved per plan? 

Per the SELI Guide version 2.0, data elements for System 
Affordability trends would pose questions such as: 

• Is the SE effort progressing towards a system that is affordable 
for the stakeholders? 

How long will the historical data need to be stored? 

Answer. The more historical data available, the more ability to 
discern problematic trends leadership will have. If file size of the 
data elements do not take up a lot of database / disk space, all of 
the data should be saved within a central repository. A 
recommendation would be to store enough to show a project’s 
trends to see improvements or declines in certain SE / SE 
management areas. 
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4. 


Data Analysis 


• Can a process be constructed to avoid data calls? 

Answer. The generic SEHV methodology, as shown in Chapter V, 
reduces manual processes within data calls by adding automated 
processes. This will reduce disruption amongst SE / SE 
management with regards to data calls. 

• What calculations need to be performed on the data elements? 

Answer. Adding values of data elements which form graphical 
trends over time is necessary for SELIs. These computed values 
should be in the form of arrays of numbers that correlate with time 
values. The x-axis would primarily be time where the y-axis would 
show the value of the planned vs. actual measure of each SELL 
Specific data element information is located within Chapter IV. 

• What trending patterns are being looked for in the data? 

Answer. Deviations from planned values can indicate potential 
issues especially when deviations are recurring. Staying within a 
set range of acceptable values would demonstrate a consistent 
well-managed project with respect to SE / SE management. 
Specific recommended methodologies for SELI trending are shown 
within Chapter IV. 


5. Data Display 

• What visualizations best represent the data? 

Answer. Line graphs and bar graphs represent the data effectively 
for SELIs. Displaying trends of SELIs to see planned vs. actual 
values would show if a project is on track. Deviations would 
trigger leadership to intervene in order to assist with getting the 
project back on track. 


6. SEHV Stakeholders 

• Who are the stakeholders that need to view these SELIs? 

Answer. Stakeholders were matched with SELIs using the 
descriptions of roles within the SSC-A CAO CONOPS and IPT 
Lead Handbook. The results are shown in Appendix C. 
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7. 


SELIs 


• What are the top SELIs that will provide the most insight to 
leadership for determining how well the SE project is going? 

Answer. The top SELIs are shown in Chapter II within the SELI 
down select process. These SELIs were matched to stakeholders 
from descriptions of roles within the SSC-A CAO CONOPS and 
IPT Lead Handbook. 

• Which SELIs do the stakeholders find most-useful? 

Answer. Using the SSC-A CAO CONOPS and IPT Lead Handbook 
as a basis for stakeholder descriptions, Appendix C depicts which 
stakeholders are aligned with SELIs. All 18 SELIs are important 
and necessary to monitor project status with respect to SE / SE 
management. 


8. Other Topics 

• What is good systems engineering / management? Is it similar to 
good project management with ensuring the SEHV system functions 
satisfy stakeholder requirements? 

Answer. Good SE / SE management practices are shown by 
applying and monitoring SE practices consistently on a project. 
This is similar to good PM by frequently performing and 
monitoring activities that affect cost, schedule, and performance 
parameters. 

• How are other similar commands monitoring SEHV? (Ex. SSC-PAC, 
NavFac, NavAir, Army Corps., AF ESC) 

Answer. NAVAIR uses SELI methodology on their AC AT II 
airplane development program. NA VAIR developed the ALI tool to 
support displaying SELIs for their program. SEHV will be used 
amongst multiple programs and projects to depict the command’s 
overall health with respect to SE / SE management. 

• How much will it cost to collect SELI data elements? 

Answer. Cost analysis of the SEHV concept is shown in Chapter 
VI. 

• How long should it take to obtain a required visualization? 

Answer. Once the SEHV system is in place, automation of manual 
processes such as data collection, and trending can support 
visualization instantly. As long as monthly use of mandated SE / SE 
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management tools are being utilized amongst projects, fresh up-to- 
date data would be available for viewing. 

How many metrics or trends should be viewed? 

Answer. Descriptions of the selected SELIs within Chapter IV 
elaborate on specific metrics and trends that should be viewed. 
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APPENDIX E. SUPPLEMENTALS 


The following items are supplemental to this report and are available at the 
Dudley Knox Library of the Naval Postgraduate School. 

• SEHV Operational Views 

• Innoslate IDEFO Models 

• SSC-A Joint Framework 

• Extends im Models 

• Cost Spreadsheet 
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